Software Framework for Analysis of Complex Microscopy Image Data

IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE, VOL. 14, NO. 4, JULY 2010 1075

A Software Framework for the Analysis of ComplexMicroscopy Image Data

Jerry Chao, E. Sally Ward, and Raimund J. Ober, Senior Member, IEEE

AbstractTechnological advances in both hardware and soft-ware have made possible the realization of sophisticated biologi-cal imaging experiments using the optical microscope. As a result,modern microscopy experiments are capable of producing compleximage datasets. For a given data analysis task, the images in a setare arranged, based on the requirements of the task, by attributessuch as the time and focus levels at which they were acquired. Im-portantly, different tasks performed over the course of an analysisare often facilitated by the use of different arrangements of theimages. We present a software framework that supports the useof different logical image arrangements to analyze a physical setof images. This framework, called the Microscopy Image Analy-sis Tool (MIATool), realizes the logical arrangements using arraysof pointers to the images, thereby removing the need to replicateand manipulate the actual images in their storage medium. Inorder that they may be tailored to the specific requirements ofdisparate analysis tasks, these logical arrangements may differ insize and dimensionality, with no restrictions placed on the numberof dimensions and the meaning of each dimension. MIATool addi-tionally supports processing flexibility, extensible image processingcapabilities, and data storage management.

Index TermsImage analysis, image viewer, microscopy, multi-dimensional data, software framework.

I. INTRODUCTION

THE optical microscope has been an invaluable tool for thestudy of biological events at the cellular, the subcellular,and more recently, the single molecule level (e.g., [1][3]). Withadvances in both hardware and software technology, the micro-scopist today is well-equipped to design and operate sophis-ticated microscopy image acquisition systems. As microscopyimaging experiments have become more creative, however, sohave the resulting image data grown in complexity. Therefore,to obtain the desired information from the acquired images in anefficient manner, software is needed that facilitates the analysisof complex image datasets. When designing such a softwareapplication, the nature of the image data and of its analysisrequirements warrants consideration.

Manuscript received April 27, 2009; revised September 28, 2009; acceptedApril 15, 2010. Date of publication April 26, 2010; date of current version July9, 2010. This work was supported in part by the National Institutes of Healthunder Grant R01 GM071048, Grant R01 AI050747, Grant R01 AI039167, andGrant R01 GM085575.

J. Chao and R. J. Ober are with the Department of Electrical Engineering,University of Texas at Dallas, Richardson, TX 75080 USA and also with theDepartment of Immunology, University of Texas Southwestern Medical Center,Dallas, TX 75390 USA (e-mail: [email protected]; [email protected]).

E. S. Ward is with the Department of Immunology, University ofTexas Southwestern Medical Center, Dallas, TX 75390 USA (e-mail:[email protected]).

Color versions of one or more of the figures in this paper are available onlineat http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TITB.2010.2049024

A. Software Design Considerations for Image Data Analysis1) Different Image Arrangements for Different Analysis

Tasks: Supported by image acquisition software and hardwarecomponents such as optical filters, focusing devices, and imagedetectors, modern microscopy experiments are capable of pro-ducing complex and multidimensional image datasets. Through-out the course of a fluorescence microscopy (e.g., [4], [5]) ex-periment, for example, images of different colors (i.e., differentwavelengths) can be captured at different times by one or morecameras at different focus levels. Depending on the nature ofan analysis task that needs to be performed, the images are ar-ranged along an appropriate number of dimensions by color,focal position, acquisition time stamp, and/or any other experi-mental or analytical parameters. Importantly, in an analysis thatcomprises different types of tasks, different arrangements of theimages may be used to facilitate the execution of the compo-nent tasks. As necessitated by the specifics of the tasks, thesearrangements may differ in the number of dimensions as wellas the meaning of the dimensions.

In the most general case, an arrangement is N -dimensional(where N is any positive integer) and the meaning of eachdimension is designated as required by an analysis task. For ex-ample, a simple linear (i.e., 1-D) arrangement with the imagessorted in no particular order may be sufficient for a visual inspec-tion of the general image quality. However, a 2-D arrangementwith the same images sorted by time in one dimension and colorin the other may be more suitable for the purpose of generatingoverlays of the different colors. In addition, an arrangement is ingeneral not limited to a reordering of the entire set of acquiredimages, but may comprise only some of the images and/or con-tain repeated images, also as necessitated by the particular taskat hand. For example, to generate the overlay of two large timelapse series acquired simultaneously, but at different rates bytwo cameras, one might choose to work with only a small timesegment of interest, and to repeat within that segment imagesfrom the slower camera to temporally align them with the im-ages from the faster camera.

We note that the integer N does not include the x and ydimensions that are intrinsic to an image. An N -dimensionalarrangement of images is, therefore, equivalent to what wouldcommonly be referred to as an (N + 2)-dimensional imagedataset.

2) Large Numbers of Images: Besides the multitude of waysin which it may be arranged, a given dataset often consists ofa large number of images. Using fast frame rate cameras, forexample, microscopists can acquire many images in a relativelyshort period of time, ending up with thousands or even tensof thousands of images by the end of an experiment. Given the

1089-7771/$26.00 2010 IEEE

1076 IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE, VOL. 14, NO. 4, JULY 2010

limited hardware resources of a conventional personal computer,data of this size poses a challenge in terms of both storage onthe hard disk and processing in random access memory (RAM).

3) Heterogeneity of Images and Flexible Processing: In gen-eral, a microscopy experiment can produce images of differentsizes. For example, when using multiple cameras with differentspecifications, one may be constrained to images of similar, butnevertheless different sizes. In addition, the use of image acqui-sition software programs from different camera manufacturersmay result in a dataset composed of images in different file for-mats. It would, therefore, be useful for a software application tosupport the display and processing of a heterogeneous datasetcomprising differently sized images of potentially different fileformats. The idea of heterogeneity is also important in termsof the processing of the image data. Given an arrangement ofimages, one should be able to process individual or subsets ofthe images differently, but at the same time also have the abilityto operate on all images uniformly. For example, due to photo-bleaching of the imaged fluorophore, one might find it necessaryto apply different pixel intensity adjustment settings to imagesin different segments of a time lapse series. On the other hand,to confine an analysis to a region of interest, one might need tocrop all the images in the exact same way.

4) Provision for Adding New Analysis Capabilities: An im-portant point to take into account in the software design is thewide variety of image analysis requirements in microscopy thatrange from simple tasks such as the cropping of an image to moresophisticated processing such as image deconvolution. Not onlyis it impractical to support all the existing image processingalgorithms for all types of analyses, it is also important to notethat analysis requirements are constantly evolving and that cus-tomized or new algorithms are always needed. Therefore, it isessential that a general microscopy data analysis applicationprovide some means for the incorporation of new capabilities.

5) Organized Storage of Images and Associated Information:A last point to consider is that, in addition to the images, thereare other types of important information that an analysis soft-ware application should maintain, potentially on a per-imagebasis. These include the processing settings (e.g., intensity ad-justment settings, crop settings, etc.) which have been appliedto the images and which can be stored to provide a history of theprocessing, the metadata (e.g., acquisition time stamp, imagedfluorophore, etc.) which are essential for certain types of analy-ses, and analytical results (e.g., computed background intensity,number of identified objects of interest, etc.) which need to bekept. In light of the various types of information that need tobe saved and associated with the images, a software applicationshould support a storage management mechanism that helpswith the organization of the images and any related experimen-tal or analytical information, both on a temporary basis in RAMand on a permanent basis on the hard disk.

B. Current Software SolutionsSoftware has been and continues to be developed by various

parties to support the analysis of microscopy image data. TheOpen Microscopy Environment [6], for example, takes an in-

formatics approach to the analysis and storage of microscopydata. This environment defines an extensible data model for themanagement of not only the images themselves, but also themetadata and the analytical results that are associated with theimages. In [7], a data model and architecture are introduced inthe context of leveraging grid technologies for the knowledge-based processing of large image datasets. Another example isthe popular Java-based application ImageJ [8], which offers anabundance of image analysis capabilities that range from stan-dard functionalities such as intensity adjustment to advancedfeatures such as object tracking. For more information on someof the currently available software packages, see [9].

The typical microscopy image analysis software package to-day, however, assumes the use of either a single or a few imagearrangements throughout the course of an analysis. Moreover,an image dataset is limited to a certain number of dimensions,commonly set to five (i.e., limited to a 3-D arrangement), andthese dimensions are fixed to represent an images x and y di-mensions, focal position along the microscopes optical (z-)axis,color, and acquisition time. Therefore, an analysis consisting ofdisparate tasks that require arrangements of higher numbers ofdimensions with arbitrary representations is in general not read-ily supported. Some applications also require the loading of anentire set of images into RAM for viewing and processing, andtherefore, have difficulty supporting the analysis of large num-bers of images. The typical software application today also doesnot readily provide for the arbitrary reordering, repetition, andsubset construction of the images in a set. In addition, the im-ages comprising a set are commonly required to have the samesize and/or file format, and the storage of processing settings,metadata, and analytical results is not generally supported by allsoftware packages. In general, the typical software solution to-day addresses some, but not all of the aspects of image analysisdiscussed in Section I-A.

C. Microscopy Image Analysis ToolIn this paper, we discuss the design of a microscopy image

analysis software framework, which takes into account all thepoints raised in Section I-A regarding the analysis of a compleximage dataset. This framework is motivated by the recognitionthat the different tasks involved in the analysis of a dataset arepotentially facilitated by different arrangements of its images.Importantly, it is based on the central idea that these arrange-ments can be achieved as different logical views of the samephysical images which may reside either in RAM or on the harddisk. In this way, a clear distinction is drawn between the logicaldatasets (i.e., arrangements), which are used for analysis, andthe actual images in RAM or on disk which are looked upononly as physical repositories of the data, and which remain un-changed throughout the course of an analysis. In addition to itsunderlying support for multiple and arbitrary logical arrange-ments of images, this framework is a generic one where norestrictions are placed on the dimensionality of an arrangement.Furthermore, the meaning of each dimension of an arrangementcan be designated arbitrarily as necessitated by the analysis taskat hand.

CHAO et al.: SOFTWARE FRAMEWORK FOR THE ANALYSIS OF COMPLEX MICROSCOPY IMAGE DATA 1077

We note that the framework we introduce here is not meantto supersede other software solutions such as those describedin Section I-B. Rather, it is an approach to microscopy imageanalysis which we find effective in dealing with the challengesposed by a complex dataset. In fact, given the wide-rangingproblem domains, objectives, and approaches that characterizethe different software solutions, it is entirely possible that ourframework may be used in conjunction with the other solutionsto make certain types of analyses more efficient.

Our software framework, which is called the MicroscopyImage Analysis Tool, or MIATool in short, realizes differentlogical arrangements of the images in a given physical datasetvia multidimensional arrays of pointers to the images. We notethat the term pointer as used here is distinct from the pointerdata type found in programming languages such as C and C++.Rather, it is used in the sense of a general, language-independentdata type that stores the address of an image that can reside inRAM or on a hard disk. For example, if in RAM, the valuestored could be a memory address. If on disk, the value storedcould be a file path.

The use of image pointers provides at least three advantages.First, it eliminates the need to physically replicate the imagesin RAM or on the hard disk in order to create different ar-rangements, and therefore, helps to save a significant amountof memory and disk space. Also, any reordering, repetition,and subset construction of the images needed to arrive at anarrangement can all be easily achieved by moving, replicating,and selectively creating or removing pointers. Second, sincepointers are typically much smaller in size than the images theyreference, a pointer array occupies considerably less RAM thanan array of actual images. Therefore, pointer arrays allow MIA-Tool to store and process large logical datasets in RAM. Third,since pointers can refer to images of different sizes and fileformats, a pointer array can naturally support the analysis of aheterogeneous dataset.

To accommodate the analysis of the images specified by anarray of pointers, the MIATool framework employs arrays ofcorresponding size and dimension to manage the various pro-cessing settings, metadata, and analytical results that are associ-ated with the pointer-referenced images. The main idea behindthese corresponding arrays is that they allow each pointer ina pointer array to be associated with its own processing spec-ification, metadata, and analytical results. Consequently, theyprovide the flexibility to process each referenced image differ-ently, while at the same time support the uniform processing ofsome or all of the referenced images through the specification ofthe same processing settings for appropriate subsets of pointersin a pointer array. Additionally, an advantage offered by corre-sponding arrays of processing settings is that they can be savedin place of the actual images that result from the processing,which are typically much larger in size than the settings. Whenthis option is used, MIATool is able to make further savings inthe usage of the limited hard disk space and at the same timepreserve a record of the processing.

Since visual feedback plays an important role in many types ofimage processing, the MIATool framework specifies an imageviewer which supports the visualization of the images refer-

enced by an N -dimensional image pointer array. In addition, itspecifies processing tools, which allow the manipulation, via agraphical user interface, of the processing settings contained inarrays that correspond to the image pointer array that is currentlydisplayed in a viewer. By interacting with a viewer to provideimmediate visual feedback whenever processing settings arechanged, these tools make possible the interactive, on-the-flyprocessing of the pointer-referenced images. While the advan-tage of immediate visual feedback renders these tools most suit-able for types of processing that require relatively little time tocomplete (e.g., intensity adjustment, cropping, etc.), such toolscan in principle be created for all kinds of image processing.Importantly, by defining standard ways of interaction betweenthe viewer, the processing tools, and the corresponding arraysof processing settings, the MIATool framework facilitates theaddition of new processing capabilities to its existing repertoire.

To help keep track of the multiple image pointer arrays andtheir corresponding arrays of settings and information that maybe used over the course of analyzing a physical image dataset,a storage management mechanism is provided by the MIAToolframework. Capable of managing storage both in RAM and onthe hard disk, this storage manager uses a hierarchical structureto associate a physical image dataset with the arrays employedfor its analysis, and to maintain the relationships among thevarious arrays. Furthermore, it serves as the standard channelthrough which the various arrays are stored and retrieved.

The remainder of this paper is organized as follows. InSection II, we give a description of the general architectureof the MIATool software framework. In Section III, we illus-trate the use of different image pointer arrays to perform dif-ferent analysis tasks on a given physical image dataset. This isdone via examples of some commonly encountered problemsin microscopy image analysis. We follow in Section IV witha discussion on how MIATool uses arrays corresponding to animage pointer array to provide flexibility in the processing of theimages it references and to maintain the metadata and analyticalresults that are associated with those images. In Section V, wepresent the MIATool image viewer and processing tools, whichtogether provide a visual, user-interactive means for workingwith an image pointer array and its corresponding arrays. InSection VI, we describe how MIATool manages the storage ofa physical image dataset and the various types of arrays that areused for its analysis. Finally, we conclude our presentation inSection VII.

II. ARCHITECTURE

The MIATool software framework comprises three principalcomponents, as shown in Fig. 1. The first component consistsof the logical N -dimensional pointer array interpretation of aphysical image dataset (see Section III), along with correspond-ing N -dimensional arrays of processing settings, metadata, andanalytical results that are used to support the analysis of thepointer-referenced images (see Section IV). These correspond-ing arrays are contained in modules, which are capable of car-rying out the actual processing of the images. The second com-ponent includes a viewer and processing tools (see Section V)which together provide a graphical user interface for the viewing


Fig. 1. Three-component MIATool software framework. The first componentprovides the underlying representation and analysis of logical image datasets us-ing multidimensional image pointer arrays and corresponding arrays of process-ing settings, metadata, and analytical results. The second component providesa graphical user interface for the viewing and the interactive, on-the-fly pro-cessing of the pointer arrays. The third component provides the RAM and diskstorage management that associates a physical image dataset with the pointerarrays and the corresponding arrays that are used for its analysis.

of the images referenced by an N -dimensional image pointerarray, and the interactive, on-the-fly processing of those imagesby way of modifying the processing settings contained in cor-responding arrays. The third component is a storage manager(see Section VI), which associates in RAM or on the hard diska physical image dataset with its potentially many pointer andcorresponding arrays.

The underlying pointer array-based representation and pro-cessing of a logical dataset, the graphical user interface forvisual, interactive data analysis, and the image and informationstorage manager therefore constitute the three main componentsof the software framework. Provided that standard protocols ofinteraction are adhered to, this architecture is amenable to theindependent development of the components. Based on this de-sign, a prototype software application, MIATool V1.1 [10], hasbeen implemented using the technical programming languageof MATLAB (The MathWorks, Inc., Natick, MA) and its imageprocessing toolbox. A preliminary introduction to the prototypeimplementation MIATool V1.1 was published in the conferencepaper [11]. In contrast, the current paper focuses on the un-derlying implementation-independent software framework, in-cluding the reference architecture that provides a blueprint forthe implementation of the design considerations delineated inSection I-A.

In the sections that follow, we make use of diagrams createdusing the standard Unified Modeling Language (UML) (e.g.,[12]) notation system to help illustrate the design of, and theinteraction between, the three main components of the MIAToolframework. These diagrams were created using the softwarepackage StarUML 5.0 (http://staruml.sourceforge.net).

III. DATA ANALYSIS USING LOGICAL ARRANGEMENTSOF IMAGES

A complex image dataset generated by a microscopy exper-iment is often subjected to multiple types of processing tasksthroughout the course of an analysis. As identified in Section I-A

Fig. 2. ImageSet and ImageSingle classes. An ImageSet contains an N -dimensional array of ImageSingles, where each ImageSingle is a pointer whichcontains, for example, the location on the hard disk at which an image resides.An ImageSingle also stores information such as the size and color type of theimage it references. Both the ImageSet and the ImageSingle support operationsfor creating pointers to images and for retrieving images via pointers.

to be an important consideration in the design of a data analysissoftware, these different tasks are often facilitated by differentarrangements of the images that may differ in size and dimen-sionality. To address this important aspect of data analysis, theMIATool software framework supports the realization of dif-ferent arrangements of a set of images as logical datasets inthe form of arrays of pointers to the images. Since the imagesphysically reside either in RAM or on disk, a pointer is simplya memory address or a file path, which uniquely identifies animage in a dataset.

By creating, manipulating, and storing arrays of image point-ers, MIATool avoids having to create or modify physical ar-rangements of the images, which would require the replicationor shuffling of the images in RAM or on the hard disk. As apointer is typically much smaller in size than the image it refer-ences, this allows MIATool to make efficient use of the limitedamount of RAM and disk space. As a result, this also enables MI-ATool to accommodate the analysis and storage of large datasets,and hence address another important design criterion.

As shown in the UML class diagram of Fig. 2, the concept ofa logical image dataset is embodied in an ImageSet. The Image-Set class contains in general an N -dimensional image pointerarray, and supports operations such as the creation of the array(importImages) and the retrieval of an actual image via a pointer(getImage). Each image pointer in the array takes the form of anImageSingle, a class which stores not only the physical location(e.g., the file path) of an image, but also information such asthe images size, color type, and file format. In addition, theImageSingle class provides operations for creating a pointer toan image (setImage) and for retrieving an image via a pointer(getImage).

The storage of image attributes in an ImageSingle is impor-tant in that it allows the ImageSingles belonging to an Image-Set to reference images of, for example, different sizes, colortypes, and file formats. This directly enables MIATool to sup-port the analysis of heterogeneous image datasets, and thereby


address the data heterogeneity design criterion. In particular,the support for heterogeneous image data allows MIATool todeal with datasets of mixed image file formats, which can easilyarise, for example, from collaborative projects involving dif-ferent imaging modalities contributed by potentially differentresearch groups (see, e.g., [13], [14]). Note that in order to real-ize support for such datasets, an implementation of the getImageoperation of the ImageSingle class should use the value of thefile format attribute to determine the appropriate image readingroutine to invoke to retrieve the referenced image from disk.

We note that in our MATLAB implementation of the Im-ageSingle class in MIATool V1.1, the address of an image ondisk is a file path that is straightforwardly stored as a string field.However, since MATLAB does not support a native pointer datatype such as that in C or C++, the address of an image in RAMis stored in two basic ways. One, the image can reside in a fieldof the ImageSingle itself, in which case its address is simplygiven by the field. Two, the image can reside in the UserDataproperty of a MATLAB figure, in which case its address isgiven by the handle of the figure. Note that these details areimplementation-specific choices, which we made in the devel-opment of MIATool V1.1. Other methods are certainly possibleand can be used in a different implementation of the framework.

Depending on the specific requirements of an analysis, animage pointer array can be of any dimensionality and can containpointers arranged to represent an arbitrary reordering of thephysical set of images. A pointer array can also contain repeatedpointers to the same image, and at the same time consist ofpointers that refer to just a subset of the images in the completedataset. In what follows, we illustrate, using concrete examplesof problems frequently faced in microscopy image analysis, theuse of different pointer arrays (and hence, different ImageSets)to analyze a physical image dataset. These examples representa small sample of the broad range of pointer arrangements thatmay be employed to carry out different analysis tasks on adataset. Importantly, though a particular example might in andof itself be a relatively simple image processing task, togetherthey demonstrate the high level of data analysis customizationthat MIATool is designed to support. (For more data analysisexamples using pointer arrangements, see [11].)

Let us consider a fluorescence microscopy (e.g., [4], [5]) livecell imaging experiment in which a cell is labeled with twodifferently-colored fluorescent dyes. The green dye is attachedto the protein of interest, and the red dye to endosomes withwhich vesicles containing the protein of interest interact. Theobjective is to track the trajectories of the fast-moving vesi-cles and observe their associations with the relatively stationaryendosomes. Accordingly, two simultaneously running camerasare used to image the same focal plane within the cell, buteach captures the fluorescence of a different dye, and writessequentially numbered images in the order of acquisition to itsown designated directory on the hard disk. In general, the twocameras may not be synchronized in time, and hence the totalnumber of images acquired by each camera may be different atthe end of the experiment. Furthermore, the images captured bythe two cameras can potentially be of different sizes and savedin different file formats.

Fig. 3. (a)(d) Logical arrangements (image pointer arrays) of different sizesand dimensionalities for performing different analysis tasks on the same set ofphysical images. An image pointer is represented by a number followed by asubscript. The number refers to the sequence number of the physical image thatis referenced by the pointer, and the subscript g or r refers, respectively, tothe green or red color of that image. These arrangements are suitable for (a)simple visual inspection of the physical images, (b) temporally synchronizedoverlay of the physical images, (c) logical extraction of an event of interest fromthe physical images, and (d) differential processing (e.g., intensity adjustment)of duplicates of the physical images. (e) 4-D arrangement which referencesimages acquired from two different focal planes. The superscripts B andT associated with the pointers refer to the bottom and top focal planes,respectively.

To visually assess the quality of the acquired images, we mayfirst want to step through and view the image files containedin the two camera output directories. For this purpose, we canconstruct an ImageSet containing a 2-D array of image pointersthat mirrors the physical arrangement of the images on disk [seeFig. 3(a)]. Specifically, this array has two rows of potentiallydifferent lengths, each containing pointers to images in a dif-ferent directory (or equivalently, from a different camera, of adifferent color) that are ordered by the sequence numbers of theimages they reference. The MIATool viewer (see Section V) canthen be used to traverse this 2-D pointer array and display thereferenced images of possibly different sizes and file formats.Conferred by pointers that can reference images with differ-ent attributes, this ability of the MIATool framework to handleheterogeneous datasets is an important advantage over softwaredesigns that restrict datasets to consist of images of uniform sizeand file format.

After the initial verification of image quality, we may want tooverlay the images from the two cameras in order to visualize theinteractions between the green vesicles and the red endosomes.To ensure that these interactions are properly observed, overlaysneed to be performed with pairs of images that were captured atapproximately the same time. Assuming that the two cameraswere not temporally synchronized, we then need to form pairsof images based on their acquisition time stamps. This can be


done using another ImageSet which contains a pointer array thatis similar to the one used for the initial visualization, but whosecolumns represent time points rather than sequence numbers[see Fig. 3(b)]. Compared with the array of Fig. 3(a), the point-ers in one row of the array of Fig. 3(b) might be shifted withrespect to the pointers in the other row as per the time stampinformation. In addition, if images of the relatively immobileendosomes were acquired at a slower rate, pointers to these im-ages may be repeated to synchronize them against the pointersto the faster acquired images of the vesicles. Given this time-synchronized pointer array, overlaid images can be generatedby moving through the columns and processing the images onepair at a time. It is important to note that instead of creating itfrom scratch, the array of Fig. 3(b) can be derived by shiftingand replicating the pointers in the array of Fig. 3(a). In general,it is often the case that the creation of a new ImageSet can bemade more efficient by deriving it from an existing ImageSet.

Now suppose that thousands of images were taken by eachcamera, and that from the viewing of the overlaid images, weidentify a small sequence of a few hundred time points whichcontain the trajectory of a representative green vesicle. To log-ically isolate this particular sequence, we can create a thirdImageSet, which contains a 2-D pointer array that referencesonly the images confined to this time frame [see Fig. 3(c)]. Asan example of how one pointer array can sometimes be easilyderived from another, the creation of this new 2-D array sim-ply requires that we keep a contiguous portion of the array ofFig. 3(b) and discard the rest. The significantly smaller pointerarray of Fig. 3(c) allows us to focus our analysis on just a partic-ular segment of the image data. A collection of such arrays canbe generated to mark all the events of interest within a largephysical dataset.

Let us now assume that the red dye used to label the endo-somes also attaches to other cellular organelles, but at muchlower quantities. Due to the significant difference in the amountof labeling, an appropriate intensity setting for viewing thestrongly labeled endosomes will not allow the weakly labeledorganelles to be seen clearly. Therefore, a different intensitysetting is needed in order to observe any potential interactionsbetween the green vesicles and the weakly labeled organelles.A method that supports the simultaneous existence of two in-tensity adjustment settings per image, and yet requires only asingle pointer array, is to stack two copies of a given 2-D array toform a 3-D array. This 3-D pointer array will, therefore, containtwo pointers to each image along its third dimension, and eachof the two pointers can be associated with a different intensitysetting (see Section IV). Fig. 3(d) shows an example of such a3-D array of duplicate pointers that has been derived by stack-ing two copies of the array of Fig. 3(c). In order to work withit, a fourth ImageSet would be created. As other arrangementsof possibly different dimensionalities can be made of the sameset of duplicate pointers, an advantage of this particular 3-Darrangement is that, for any given image, one can go from oneintensity setting to the other by simply toggling the value of thethird dimension.

To illustrate the concept of logical datasets, we have thus fardemonstrated MIATools use of different image pointer arrays to

perform different analysis tasks on a relatively simple physicalimage dataset. To show how easily a change to the experimentcan make the resulting data more complex, let us assume thattwo additional cameras were used to capture the same greenand red fluorescence, but from a focal plane within the cellthat is located higher along the microscopes z-axis. Due tothe additional view of the cell that is provided by the imagesfrom this top plane, this multifocal plane imaging setup [15]allows us to detect the fast-moving green vesicles of interest atlocations along the z-axis that we would otherwise not be able todetect by imaging at only the bottom focal plane. Therefore,by using four cameras to simultaneously capture images fromtwo distinct focal planes, we can better visualize the trajectoriesof the green vesicles in three dimensions.

Since MIATool places no limits on the number and meaningof the dimensions of an image pointer array, the same analysistasks can be performed on the more complex dataset using thesame type of logical arrangements as before, but with an addi-tional dimension to distinguish images from the top and bot-tom focal planes. At the end of the same sequence of tasks, wewould obtain a 4-D pointer array as depicted in Fig. 3(e). Notethat in Fig. 3(e), the two 3-D arrays corresponding to the topand bottom focal planes are intended to represent a 4-D arraywhere the fourth dimension allows us to toggle between imagesof the same color and from the same time point, but from differ-ent focal planes and with potentially different intensity settings.

In practice, we have used the same types of logical arrange-ments to analyze complex image data of a similar nature. In [16]and [17], for example, 3-D trajectories of vesicles and singlemolecules inside live cells were determined from images ac-quired with multifocal plane imaging setups. In those experi-ments, multiple cameras were operated at different speeds tocapture images of different colors from up to four distinct focalplanes.

To close this section, we provide some concrete numbersregarding the size of the image dataset on which our exampleshave been based. These numbers are representative of the actualimage data we analyzed in [16] and [17] using our prototypeimplementation of MIATool, and will illustrate the significantadvantage gained with the use of image pointer arrays in termsof RAM usage.

Suppose that 6000 16-bit grayscale images, each of 420 400pixels, are acquired by the camera which captures the fluores-cence of the green dye from the bottom focal plane. Thisamounts to approximately 328 kB per image, and approximately1.88 GB for all 6000 images. At the same time, suppose 400016-bit grayscale images, each of 420 420 pixels, are acquiredby the camera that captures the fluorescence of the red dye fromthe bottom focal plane. This equates to approximately 345 kBper image, and approximately 1.31 GB for all 4000 images. Thetotal size for all 10 000 images from the bottom focal planeis then approximately 3.19 GB. Now suppose that the sameamounts of data are acquired by the two cameras that image thetop focal plane. The size of the entire set of 20 000 imagesthen becomes approximately 6.38 GB.

Given a conventional personal computer with 1, 2, or even4 GB of RAM, software applications that require the loading of


all 6.38 GB of images into RAM would either not be able to han-dle this dataset, or at a minimum cause a substantial slowdownof the application itself and the rest of the computer system.Using the pointer array representation of MIATool, however, anImageSet containing a pointer array that references the 20 000images would require a significantly smaller amount of RAM.In MIATool V1.1, for example, an ImageSet containing a 3-Darray of pointers to the 20 000 images would only take up ap-proximately 145 MB of RAM. This 3-D array is constructed bystacking two copies of the type of 2-D pointer array depicted inFig. 3(a) along a third dimension. One copy contains pointersthat reference the 10 000 images from the bottom focal plane,and the other contains pointers that reference the 10 000 imagesfrom the top focal plane.

We note that the significant savings in RAM provided by anImageSet will translate to similar savings in disk space. There-fore, instead of physically replicating images to form variousarrangements such as those illustrated in Fig. 3, the saving ofImageSets containing pointer arrangements could save a non-trivial amount of disk space. Also, it is important to point outthat, regardless of how large or small the dataset, a softwareapplication that requires all images to be of the same size wouldnot readily support the analysis of the type of heterogeneousdata described.

IV. CORRESPONDING ARRANGEMENTS OF PROCESSINGSETTINGS, METADATA, AND ANALYTICAL RESULTS

In Section III, we illustrated MIATools use of different imagepointer arrays to facilitate the different processing tasks thatcomprise a data analysis. Here, we describe the means by whichthe framework supports the execution of a given task. In general,a task may require that each of the images referenced by itspointer array be processed differently. Moreover, the processingof the referenced images may potentially be based on metadatathat is specific to each image, and may produce analytical resultsthat need to be maintained on a per-image basis. Specified as acrucial design consideration in Section I-A, MIATool supportsthis processing flexibility by making use of arrays of processingsettings, metadata, and analytical results which are constructedin parallel to an image pointer array. In what follows in thissection, we give our main focus to arrays of processing settings,but end with a discussion on arrays of metadata and analyticalresults, which can conceptually be seen as simple special casesof arrays of processing settings.

Given an N -dimensional image pointer array, one should onthe one hand be able to process each of the referenced imagesdifferently, and on the other hand have the option to processall referenced images in a uniform way. In order to accommo-date both extremes and all permutations in between, MIATooluses arrays corresponding in size and dimension to a pointerarray to support the flexible processing of its referenced images.More specifically, an image referenced by a pointer belonging toelement (x1 , x2 , . . . , xN ) of a pointer array is processed accord-ing to the settings that are stored in element (x1 , x2 , . . . , xN )of a corresponding array. With such a parallel design, customprocessing is possible on a per-image basis, while at the same

Fig. 4. Examples of Set and Single classes for the processing of ImageSetsand ImageSingles. Analogous to the ImageSetImageSingle relationship, a Setcontains an N -dimensional array of its associated type of Singles. A Singleat a particular position in the N -dimensional array stores the settings for theprocessing of the ImageSingle located at the same position in the array containedin a corresponding ImageSet. An IntensitySingle and a CropSingle contain,respectively, the settings for the intensity adjustment and cropping of the imagereferenced by a corresponding ImageSingle. Similarly, a SegmentationSingleand a LabelSingle store, respectively, settings for the partitioning and labelingof the referenced image.

time uniform processing for all or subsets of the images can beachieved by simply specifying identical settings in the appro-priate elements of a corresponding array.

Different corresponding arrays of processing settings are usedfor different processing tasks. Just as an image pointer array isstored in and managed by an ImageSet, these correspondingarrays are kept in and handled by various Set classes, as il-lustrated in Fig. 4. Arrays that store intensity adjustment andcrop settings, for example, reside in the classes IntensitySet andCropSet. Moreover, analogous to image pointers being instancesof the ImageSingle class, each element of a corresponding ar-ray is an instance of a Single class that contains the processingsettings for a specific image. As shown in Fig. 4, an Intensi-tySingle, for instance, keeps information such as the intensityadjustment method to use and the values of the associated ad-justment parameters. Similarly, a CropSingle stores parameterswhose values describe the region of the image to retain whilethe rest is trimmed.

Importantly, Fig. 5 shows that all Set classes support a com-mon repertoire of operations specified by the interface SetUsage.In addition to operations for the storing and retrieving of pro-cessing settings to and from the elements of the array that ismanaged by a Set, this interface requires a Set class to imple-ment two important operations. Given an image pointer arrayby way of an ImageSet, the initializeSingleArray operation cre-ates an array of processing settings that corresponds in size anddimension to that pointer array, and hence initializes the Setfor working with the supplied ImageSet. The applyParametersoperation accepts the same ImageSet as input, and processeseach of its referenced images according to the settings stored inthe Set. The applyParameters operation shows that Sets do notonly store the processing settings, but also carry out the actualtasks of intensity adjustment, cropping, segmentation, labeling,etc. In an analogous manner, all Single classes implement theinterface SingleUsage, which specifies analogous operations atthe level of a single image.


Fig. 5. SetUsage and SingleUsage interfaces. The SetUsage interface is sup-ported by all Sets and specifies standard Set operations such as the creation of anN -dimensional array of Singles, the assignment and the retrieval of processingsettings to and from a particular Single, and the actual processing of the imagesreferenced by a corresponding ImageSet according to the stored processing set-tings. Analogously, the SingleUsage interface is implemented by all Singles andspecifies similar standard operations at the level of a Single.

In general, the output of the applyParameters operation ofa Set is a new ImageSet containing an array of pointers to the(e.g., intensity-adjusted, cropped, segmented, or labeled) imagesgenerated by its processing. These resulting images can againreside either in RAM or on the hard disk, and if need be, they canbe subjected to further processing by the next Set in a sequenceof processing operations. In this model of processing, actualimages and an ImageSet that refers to them are generated ateach intermediate processing step. While this is a good way toproceed in cases where images created at intermediate steps aredesired, in other cases it could present problems when there isinsufficient RAM or disk space.

An alternative model of processing is therefore to iteratethrough the initial array of pointers and process one referencedimage at a time from beginning to finish. In this way, only asingle set of final images is created at the end of the processingsequence. This alternative approach can be readily realized bycarrying out the processing at the level of Singles instead of Sets.That is, given an image (i.e., an ImageSingle) from the initialImageSet, we can process it from beginning to end by invokingin proper order the applyParameters operation on each of itscorresponding Singles. The two models of processing need notbe mutually exclusive, and depending on the available hardwareresources and the intermediate results that are desired, they canbe applied as appropriate to different segments of a sequence ofprocessing steps.

In addition to processing flexibility, the use of correspond-ing arrays provides at least three further advantages. First, theparallel design allows the straightforward propagation of pro-cessing settings when one image pointer array is derived fromanother as illustrated by the sequence of analysis tasks described

in Section III. Due to the one-to-one correspondence betweena pointer array and its associated arrays of processing settings,the arrays of settings corresponding to an existing pointer ar-ray can be manipulated in exactly the same way as the pointerarray to arrive at arrays of settings that not only correspond insize and dimension to a derived pointer array, but also retainthe same processing settings for the images referenced by thederived pointer array. This carryover of processing settings isuseful since in many situations, pre-existing settings apply justas well to a derived image pointer array.

Second, by adhering to the paradigm of Sets and their as-sociated Singles, the design criterion of software extensibilityis accounted for as new image processing capabilities can beincorporated into MIATool with relative ease in the form ofnew types of Sets and Singles. As we will see in Section V,this paradigm also forms the basis for the extensibility of theimage display and the interactive, on-the-fly processing frame-work adopted by the MIATool viewer and the various imageprocessing tools. Third, the saving of Sets and Singles providesa practically useful alternative to the saving of the images thatresult from the processing. A Single that contains processingsettings is typically much smaller than the image that resultsfrom the processing, and therefore, occupies significantly lessdisk space. This is another way by which MIATool addressesthe design consideration of accommodating the analysis of largeimage datasets. Furthermore, the saved settings readily providea record of the processing that can be used at a later time togenerate the desired images.

Besides processing settings, corresponding arrays can be usedfor the storage of information such as the metadata and the ana-lytical results that are associated with the images referenced bya pointer array. For example, new types of Sets and Singles canbe created to maintain for each referenced image of a pointer ar-ray metadata such as its acquisition time stamp, focus level, andcolor (i.e., wavelength). Analytical results such as the objects ofinterest identified in a tracking application and their computedattributes (e.g., size, centroid, fluorescence intensity, etc.) canalso be stored on a per-image basis using the Set and Singleparadigm. Note that since these Sets and Singles are used purelyfor information maintenance and do not perform any process-ing on images, the applyParameters operations specified by theSetUsage and SingleUsage interfaces can be trivially defined toeither do nothing or to simply return the stored information.

V. IMAGE VIEWER AND PROCESSING TOOLS

In Section IV, we discussed MIATools use of correspondingarrays (i.e., Sets) of processing settings, metadata, and analyticalresults for the processing of the images referenced by an array ofpointers (i.e., an ImageSet). There, the nature of the processingdescribed assumes that the precise processing specifications arealready known, and that therefore no visual feedback or user in-teractivity is necessary during the execution of an analysis task.For many types of image processing, however, the ability tovisualize the images as well as the changes made to the imagesis desirable if not crucial. A simple intensity adjustment, for ex-ample, often requires the user to manually try out and visually


Fig. 6. Instance of the MIATool V1.1 viewer that has been opened with foursliders for the traversal of a 4-D image pointer array. The array contains repli-cated pointers that reference a physical dataset of 11 321 images on the harddisk, acquired using a two-plane multifocal plane microscopy imaging setup.The dimensions of the pointer array are given by 2 (focal planes) 3946 (timepoints) 2 (colors/fluorophores) 2 (intensity settings). The displayed imageis an RGB overlay, formed on the fly, of two grayscale images of a human mi-crovascular endothelial cell acquired at different wavelengths corresponding toQD 655-labeled IgG (red channel) and pHluorin-labeled FcRn (green channel).The two grayscale images were acquired by two cameras that simultaneouslyimaged the bottom focal plane of the two-plane imaging setup.

assess different adjustment methods and/or intensity settings be-fore deciding on the best choice. In this section, we describe theviewer and tools specified by the MIATool framework to supportimage visualization and user-interactive, on-the-fly processingwith visual feedback.

As we alluded to in the discussion of Fig. 3(a), perhaps themost basic of necessities when given an N -dimensional imagepointer array is to be able to traverse the array and view the ref-erenced images. To address this requirement, MIATool providesas a basic component of its graphical user interface an imageviewer that supports the traversal and display of the images refer-enced by a pointer array. Since each pointer in an N -dimensionalarray is uniquely identified by an N -tuple (x1 , x2 , . . . , xN ), theMIATool viewer allows the selection of a pointer with a set of Ncontrols such as sliders, each specifying the value of a differentdimension. The screen capture of Fig. 6, for example, shows aninstance of the MIATool V1.1 [10] viewer that has been openedwith four sliders for the traversal of a 4-D image pointer array.

Upon the selection of a pointer, the MIATool viewer retrieveson the fly the referenced image from RAM or the hard diskand displays it to a window. The currently displayed image isalways overwritten with a newly retrieved image. In this way,the viewing of a large image dataset which physically resideson disk is made possible without having to first load all thereferenced images into RAM, which can often be a problemwhen the amount of RAM is very limited. The physical imagedataset displayed by the viewer of Fig. 6, for example, consistsof 11 321 16-bit grayscale images, each of 320 390 pixels. Thesize of each image is therefore approximately 244 kB, and thatof the entire dataset is approximately 2.63 GB. The loading ofall 2.63 GB of data into RAM could already prove difficult witha conventional personal computer. However, as we explain inthe following, the 4-D image pointer array (i.e., ImageSet) that isloaded in the viewer of Fig. 6 actually references 31 568 images

by virtue of replicated pointers to the 11 321 physical imageson the disk. Whereas this 4-D ImageSet as implemented inMIATool V1.1 only takes up approximately 231 MB of RAM, anequivalent 4-D arrangement of actual replicated images wouldrequire around 7.34 GB of RAM.

The set of 11 321 physical images was produced by the type oflive cell fluorescence imaging experiment we carried out in [17],where multifocal plane microscopy [15] was used to image andtrack in three dimensions the itineraries of the neonatal Fc re-ceptor (FcRn) and its ligand immunoglobulin G (IgG) in a hu-man microvascular endothelial cell. Similar to the experimentdescribed in Section III, four cameras were used to simultane-ously acquire time sequences of images from two distinct focalplanes, and the images from each camera were written to thecameras own separate directory in the order they were acquired.In the bottom focal plane, the first camera captured the fluo-rescence from pHluorin-labeled FcRn, and the second cameracaptured the fluorescence from quantum dot (QD) 655-labeledIgG. In the top focal plane, the third camera captured the flu-orescence from monomeric red fluorescence protein (mRFP)-labeled FcRn, and the fourth camera captured the fluorescencefrom QD 655-labeled IgG. (For more details concerning theexperiment, see [17].)

Since the four cameras acquired images at different rates,the resulting dataset of 11 321 images consists of four directo-ries of image sequences of different lengths. This 2-D physicalarrangement of images is, therefore, sorted by camera and se-quence number, and the images acquired by the different cam-eras are not temporally synchronized with one another. How-ever, to properly visualize the trajectories of, and the interactionsbetween FcRn and IgG, we needed to view overlays of tempo-rally synchronized images from each focal plane with differentintensity settings. To this end, we made use of the pointer arraymanipulations illustrated in Fig. 3 to temporally synchronize theimages and introduce the necessary dimensions. The resulting4-D pointer array is the 2 (focal planes) 3946 (time points)2 (colors/fluorophores) 2 (intensity settings) array loaded inthe viewer of Fig. 6, and it is similar to the 4-D array depictedin Fig. 3(e).

Abstracted by the class MIAToolViewer as shown in Fig. 7,the MIATool viewer supports various display modes that areuseful for microscopy image analysis. Besides the standard 2-D display, an image can be presented, for example, as a 3-Dmesh or as one of the color channels in an overlay (as in Fig. 6)with other images. Note that modes such as the overlay dis-play require the selection of multiple images instead of just one,and the mechanism for making such a selection in a viewer isimplementation-specific. In MIATool V1.1, for example, view-ing of RGB overlays is achieved by scrolling through one di-mension of an ImageSet and viewing the images in anotherdimension in groups of up to three as different color channels ofa single RGB image. In Fig. 6, the dimension from which im-ages are taken to form the overlays is indicated by the disabled(i.e., grayed out) slider.

It is important to point out that the specification of the si-multaneous display of multiple images in a MIAToolViewer isa general concept that readily includes the 3-D visualization of


Fig. 7. MIAToolViewer class. A MIAToolViewer supports the interactive traver-sal and viewing of the images referenced by an ImageSet. Optionally, it usesthe processing settings contained in the various types of Sets to process (e.g.,intensity-adjust, crop, etc.) the images on the fly before displaying them in pro-cessed form. A MIAToolViewer supports operations for loading the ImageSetto view and the Sets to use, and for displaying the currently selected images.Different types of displays can be specified. A grayscale image, for example,can be visualized individually as a standard 2-D image, a 3-D mesh, or a contourplot, or it can be displayed simultaneously with other grayscale images, eitherin parallel but in its own window, or as a color channel in an overlay.

the images. Just as up to three images can be selected to forman RGB overlay, up to M (where M is an integer greater than 1)images can be selected and displayed as a 3-D volume of, forinstance, a time lapse sequence or a z-stack. Such visualizationoptions in three dimensions can be useful, if not essential, in theanalysis of complex biological structures (see, e.g., [18], [19]).

Importantly, Fig. 7 also illustrates that, given an image pointerarray in the form of an ImageSet, the viewer can additionallybe supplied with corresponding Sets. By carrying out on-the-fly processing of an image according to its associated settingscontained in these Sets, the viewer enables the viewing of pro-cessed (e.g., intensity-adjusted, cropped, segmented, etc.) im-ages without requiring that they pre-exist in RAM or on thehard disk. However, in order that the viewer is shielded from thespecifics of the various processing tasks, this on-demand pro-cessing of images relies on its interaction with the SetUsage andSingleUsage interfaces (see Fig. 5). As such, new processingcapabilities can be added to the viewer by way of new types ofSets and Singles that support operations which conform to theirrespective interfaces.

To allow the user to view and to interactively specify andmodify processing settings, MIATool provides graphical userinterfaces for displaying and manipulating the contents of thevarious types of Sets. The graphical user interface to each typeof Set is managed by a different image processing tool. For ex-ample, as shown in Fig. 8, an IntensityTool is responsible formediating access to an IntensitySet, while a CropTool is the in-termediary that facilitates the manipulation of a CropSet. As itis important for the user to receive immediate visual feedbackon the effects of the changed settings on the images, an imageprocessing tool is designed to be able to work with the MIA-Tool viewer. Through the interface ViewerUsage (see Fig. 8)that is supported by the viewer, a Tool can retrieve a Set fromthe viewer, return to it a modified version of the Set based onthe user input, and ask the viewer to reprocess and redisplay

Fig. 8. Interaction of the MIATool viewer with various image processing toolsvia the ViewerUsage and the ToolUsage interfaces. The processing settingscontained in a Set can be displayed and modified through the graphical userinterface provided by a corresponding Tool. The settings in an IntensitySet,for example, can be manipulated using an IntensityTool. Via the ViewerUsageinterface supported by the MIATool viewer, any type of Tool can retrieve itscorresponding Set, return a potentially modified version of the Set, and forimmediate visual feedback request the viewer to redisplay the current imagewhich may have been altered by the changed settings. On the other hand, alltypes of Tools implement the ToolUsage interface which allows the viewer to,for example, open and close a Tool, and to request a Tool to update its settingsdisplay whenever a new current image is selected.

Fig. 9. Instance of the MIATool V1.1 intensity adjustment tool (upper rightpanel) and an instance of the crop tool (lower left panel) that have been openedfor the display and modification of, respectively, an IntensitySet and a CropSetthat are loaded in an instance of the MIATool viewer (upper left panel). Theviewer instance is the same as the one in Fig. 6 and is displaying a version of thesame image that has been intensity-adjusted and cropped on the fly accordingto altered intensity and crop settings specified via the two tools. The settingsdisplayed in the tools reflect that of the modified image currently displayed.

the current image. Any changes due to the modified settings arethen reflected immediately in the refreshed display. As an ex-ample, Fig. 9 shows the same viewer as in Fig. 6, but displayingan altered version of the same image that has been specifiedinteractively via the intensity adjustment tool and the crop toolshown.


Conversely, all image processing tools implement a commoninterface ToolUsage (see Fig. 8), which is used by the MIAToolviewer. Relying on the operations specified by this interface, theviewer can, for example, open and close the graphical user inter-face that is provided by a Tool without knowing the Tools imple-mentation details. Through this interface, it can also request thevarious Tools to display the processing settings that correspondto the currently displayed image (e.g., the settings displayed inthe intensity adjustment tool and the crop tool of Fig. 9 reflectthat of the displayed image). This capability is essential as theviewer uses it to ensure that the information reported by the toolgraphical user interfaces stays updated whenever a new currentimage is selected by the user. Importantly, by adhering to theToolUsage and ViewerUsage interfaces, the design criterion ofsoftware extensibility specified in Section I-A is also fulfilledfor the visual and interactive component of MIATool as viewer-compatible image processing tools can be created with relativeease to support new types of Sets and Singles.

Along with the Sets they modify, the image processing toolswe have discussed thus far constitute a simple and extensiblemeans of supporting processing on a per-image basis. While itmakes sense to realize many kinds of image processing in thismanner, there is a different category of processing that operateson multiple images at a time. Just as 3-D visualization displaysmultiple images together in one form or another, 3-D processingsuch as movie making and particle tracking operates on multipleimages at a time. Analogous to how 3-D visualization can be re-alized, an implementation of the MIATool framework can takeadvantage of the MIATool viewers multiple image selectionfeature to create 3-D processing tools. Internally in our labora-tory, for example, a movie making tool and other types of 3-Dprocessing tools have been implemented which, analogous to theway RGB overlays are displayed in MIATool V1.1, operate onimages along a particular dimension of an image pointer array.

VI. STORAGE MANAGEMENT OF IMAGES AND ASSOCIATEDINFORMATION

In Section III, we gave practical examples of microscopy im-age analysis which illustrate that different image pointer arrays(ImageSets) may be employed for performing various analysistasks on images from the same physical dataset (see Fig. 3).In Section IV, we discussed corresponding arrays of settings ofvarious types (IntensitySets, CropSets, etc.), as well as arrays ofmetadata and analytical results, which may be associated witha given image pointer array. Consequently, a physical set of im-ages in RAM or on the hard disk can be associated with severalImageSets, each of which can in turn be associated with sev-eral corresponding Sets. All things combined, a physical imagedataset can potentially be associated with many ImageSets andcorresponding Sets of different types. Even more Sets could beinvolved if, for example, multiple Sets of the same type are asso-ciated with the same image pointer array. One can imagine, forinstance, the use of multiple CropSets to define different regionsof interest within the referenced images.

In addition to being potentially large in quantity, the Sets as-sociated with a physical dataset are related to one another in

Fig. 10. MIATools image and Set storage management. (a) Sketch of a repre-sentative hierarchical directory structure used by the MIATool storage manager.Underneath a root directory are two subdirectories, one containing the physicalset of images and the other the logical datasets used to analyze those images.Within the latter subdirectory, each logical dataset (i.e., ImageSet) occupies itsown subdirectory. Underneath each ImageSet subdirectory are subdirectorieswhich store the Sets (e.g., IntensitySets, CropSets, etc.) that correspond to thatImageSet. (b) Storage manager abstracted by the MIAToolDirectory class. AMIAToolDirectory manages a directory structure like the one depicted in (a). Itkeeps track of the location of the root directory and the names of all of its sub-directories, and maintains information such as the number of saved ImageSets,their file names, and similar details pertaining to any saved Sets correspondingto each ImageSet. A MIAToolDirectory also supports operations for the savingand retrieval of the various Sets to and from the directory structure it manages.

different ways. For example, while all ImageSets are peers inthe sense that they represent independent logical datasets de-rived from the same physical image data, a given IntensitySet isattached to the particular ImageSet with which it is associated.

Due to the potentially large amount of differently relatedinformation that can be associated with a physical image dataset,a storage management mechanism that provides organizationis needed as pointed out in Section I-A as a software designcriterion. That is, not only is it important for this mechanism togroup data and information that are related, it is essential that itorganizes them in a way that encodes their relationships. To thisend, MIATool employs a manager which enforces storage in ahierarchical directory structure to help with the organization of aphysical image dataset with the potentially many and differentlyrelated ImageSets and corresponding Sets that are used for itsanalysis. The hierarchical directory structure importantly allowsthe manager to capture the relationships between the variousImageSets and corresponding Sets, and it can be applied to bothstorage in RAM and storage on the hard disk.

As illustrated in Fig. 10(a), the MIATool storage managerstores all ImageSets and their corresponding Sets in the samedirectory as the physical image data with which they are asso-ciated, but under a subdirectory structure of their own to denotea clear separation between the physical images and the logical


datasets used to analyze them. Within this subdirectory, eachImageSet is given its own subdirectory, underneath which all ofits corresponding IntensitySets, CropSets, etc., are grouped bytype and stored in separate subdirectories of their own. For stor-age on disk, this hierarchy of directories is literally created onthe hard disk. For storage in RAM, however, the implementeddirectory structure would only be logical in nature.

The MIATool storage manager is abstracted by the class MI-AToolDirectory, diagrammed in Fig. 10(b). An instance of MIA-ToolDirectory acts as a table of contents for the directory struc-ture it manages. It keeps track of information such as the locationof the top level directory and the names of the subdirectoriesthat contain the physical image data and the ImageSets. It alsorecords information such as the number of saved ImageSets andtheir file names, and maintains details such as the number andfile names of corresponding Sets of each type that are associatedwith each ImageSet. (Note that when the storage is in RAM, de-tails such as directory and file names are still relevant as theyprovide a means for uniquely identifying each ImageSet andits associated Sets.) Importantly, a MIAToolDirectory providesoperations for the saving and the retrieval of ImageSets and thevarious Sets to and from its managed directory structure. Theseoperations ensure that ImageSets and their associated Sets aresaved to and retrieved from the correct locations within the hi-erarchy of directories, and that the information contained in aMIAToolDirectory is updated properly (e.g., that an appropriatecounter is incremented when a new Set is saved).

Though most straightforwardly interpreted and implementedas a directory structure that resides on a single hard drive ofa single computer, it is important to note that the hierarchicalstructure managed by a MIAToolDirectory can be realized asone that spans multiple networked computers running poten-tially different operating systems. As long as all the images andall the associated ImageSets and their corresponding Sets areuniquely identifiable and retrievable across the network and thevarious platforms, a MIAToolDirectory can be implemented thatmanages subdirectories of images, ImageSets, and other Setsthat reside on different computers. A MIAToolDirectory imple-mentation that supports such network-spanning, cross-platformimage datasets can be particularly useful for large-scale collab-orative projects (see, e.g., [13], [14]).

In a collaborative project, it is also important that differ-ent users are able to access the same image dataset simultane-ously, and yet manipulate it differently. This sharing of data canhelp to avoid the replication of the large amounts of data thatare especially typical of large-scale collaborations. To this end,read-only image data can be stored on shared drives across thenetwork, such that multiple users can access the images at thesame time, but are not able to overwrite them. Given a shareddataset, two general approaches can be used to support the anal-yses performed by the users. With either approach, each user cancreate and work with his or her own ImageSets that referencethe same shared images. However, the two schemes differ in theway the access to the shared images and the storage of resultsare realized.

In the first approach, each user has his or her own storagemanager (i.e., MIAToolDirectory) through which he or she ac-

cesses the shared images and saves the results of analysis. In thisscenario, each users own ImageSets and corresponding Sets aresaved under his or her own directory structure. This approachavoids the sharing of a storage manager by the users, but lacksa central mechanism that keeps track of the storage locations ofthe results of all analyses that are associated with the shared im-age data. (Note that since they are read only, it is possible to havemultiple storage managers that provide potentially concurrentaccess to the shared images.)

In the second approach, all users go through a shared storagemanager to access the shared images and save their results.In this scheme, each users ImageSets and corresponding Setsare saved under a single central directory structure. The singlestorage manager adopted by this approach provides a means tocentrally locate the results of all analyses that are performed onthe shared image data. However, while the storage manager canallow concurrent access to the read-only images, the saving ofthe users analysis results must be done in a sequential way toensure that the manager always has accurate knowledge of thecontents of the central directory structure.

VII. CONCLUSIONWe have described the MIATool software framework which

has been built based on several design considerations pertainingto the analysis of complex image datasets produced by modernoptical microscopy experiments. A central design criterion issupport for the use of different arrangements of a set of imagesto facilitate the execution of the different processing tasks thatcomprise a microscopy data analysis. To this end, MIATool sup-ports data analysis that is based on logical image arrangementsin the form of arrays of pointers to the physical images. Theseimage pointer arrays can be of arbitrary size and dimension, thusallowing MIATool to accommodate analysis tasks with disparaterequirements.

The use of image pointer arrays also allows MIATool tosupport the storage and analysis of large image datasets, andthereby address another important software design considera-tion. Pointer arrays are typically significantly smaller in sizethan the sets of images they reference. Therefore, by enablingthe realization, manipulation, and storage of different image ar-rangements without the need to replicate and shuffle the actualimages in their physical storage medium, pointer arrays makefor the space-efficient usage of RAM and the hard disk, andnaturally permit MIATool to handle datasets containing largenumbers of images. An additional advantage of using a pointerarray is that its pointers can refer to images of different sizesand file formats. Consequently, MIATool can easily support setsof different-sized images of possibly different file formats, andhence account for the data heterogeneity design consideration.

To address the design criterion of flexibility in processing, theidea of image pointer arrays is complemented by correspondingarrays of processing settings, metadata, and analytical resultswhich provide the ability to perform differential processing ona per-image basis. Importantly, the construction of these corre-sponding arrays is described by a simple paradigm that, whenadhered to, allows the relatively easy incorporation of new image


processing capabilities. A crucial design consideration, the ideaof software extensibility also plays an important role in thedesign of MIATools image viewer and processing tools. Theviewer supports the visualization of the images referenced by amultidimensional image pointer array, while the tools supporttheir interactive, on-the-fly processing via the modification ofthe processing settings stored in corresponding arrays. By spec-ifying the viewer and the processing tools to interact throughwell-defined interfaces, the framework allows the straightfor-ward addition of new viewer-compatible processing tools.

Last, in accordance with the storage management design cri-terion, MIATool specifies a storage manager which enforces,either in RAM or on the hard disk, the association of a physicalimage dataset with the pointer and corresponding arrays that areused for its analysis. Importantly, this manager plays an orga-nizing role in using a hierarchical directory structure to maintainthe relationships among the various arrays.

The MIATool framework and its current implementation [10]have been developed over the course of several years basedon design elements we have found to be essential for work-ing with microscopy image data. In our laboratory, it has been,and continues to be, employed for projects of varying sophis-tication. Taken together, we find that the various features ofMIATool make it a suitable software framework for a researchenvironment where microscopy imaging experiments produceconstantly evolving data analysis requirements.

REFERENCES

[1] A. J. Lacey, Ed., Light Microscopy in Biology: A Practical Approach, 2nded. Oxford, U.K.: Oxford Univ. Press, 1999.

[2] S. Inoue and K. R. Spring, Video Microscopy: The Fundamentals, 2nd ed.New York: Plenum, 1997.

[3] W. E. Moerner, New directions in single-molecule imaging and analysis,Proc. Natl. Acad. Sci. U.S.A., vol. 104, no. 31, pp. 1259612602, 2007.

[4] B. Herman, Fluorescence Microscopy, 2nd ed. Oxford, U.K.: BIOS Sci-entific Publishers, 1998.

[5] J. W. Lichtman and J. Conchello, Fluorescence microscopy, Nat. Meth-ods, vol. 2, no. 12, pp. 910919, 2005.

[6] J. R. Swedlow, I. Goldberg, E. Brauner, and P. K. Sorger, Informatics andquantitative analysis in biological imaging, Science, vol. 300, no. 5616,pp. 100102, 2003.

[7] W. M. Ahmed, D. Lenz, J. Liu, J. P. Robinson, and A. Ghafoor, XML-based data model and architecture for a knowledge-based grid-enabledproblem-solving environment for high-throughput biological imaging,IEEE Trans. Inf. Technol. Biomed., vol. 12, no. 2, pp. 226240, Mar.2008.

[8] W. S. Rasband. ImageJ. (19972010). [Online]. Available:http://rsb.info.nih.gov/ij.

[9] J. L. Clendenon, J. M. Byars, and D. P. Hyink, Image processing softwarefor 3D light microscopy, Nephron Exp. Nephrol., vol. 103, no. 2, pp. e50e54, 2006.

[10] MIATool V1.1. (20042010). [Online]. Available:http://www4.utsouthwestern.edu/wardlab/miatool.

[11] J. Chao, P. Long, E. S. Ward, and R. J. Ober, Design and applicationof the Microscopy Image Analysis Tool, in Proc. IEEE Eng. Med. Biol.Workshop, Dallas, TX, 2007, pp. 9497.

[12] J. Schmuller, Sams Teach Yourself UML in 24 Hours, 3rd ed. Indianapo-lis, IN: Sams Publishing, 2004.

[13] J. W. Lichtman and J. R. Sanes, Ome sweet ome: what can the genometell us about the connectome?, Curr. Opin. Neurobiol., vol. 18, no. 3,pp. 346353, 2008.

[14] M. Helmstaedter, K. L. Briggman, and W. Denk, 3D structural imagingof the brain with photons and electrons, Curr. Opin. Neurobiol., vol. 18,no. 6, pp. 633641, 2008.

[15] P. Prabhat, S. Ram, E. S. Ward, and R. J. Ober, Simultaneous imaging ofdifferent focal planes in fluorescence microscopy for the study of cellular

dynamics in three dimensions, IEEE Trans. Nanobiosci., vol. 3, no. 4,pp. 237242, Dec. 2004.

[16] P. Prabhat, Z. Gan, J. Chao, S. Ram, C. Vaccaro, S. Gibbons, R. J. Ober,and E. S. Ward, Elucidation of intracellular recycling pathways leadingto exocytosis of the Fc receptor, FcRn, by using multifocal plane mi-croscopy, Proc. Natl. Acad. Sci. U.S.A., vol. 104, no. 14, pp. 58895894,2007.

[17] S. Ram, P. Prabhat, J. Chao, E. S. Ward, and R. J. Ober, High accuracy3D quantum dot tracking with multifocal plane microscopy for the studyof fast intracellular dynamics in live cells, Biophys. J., vol. 95, no. 12,pp. 60256043, 2008.

[18] E. Jurrus, M. Hardy, T. Tasdizen, P. T. Fletcher, P. Koshevoy, C.-B. Chien,W. Denk, and R. Whitaker, Axon tracking in serial block-face scanningelectron microscopy, Med. Image Anal., vol. 13, no. 1, pp. 180188,2009.

[19] J. Lu, J. C. Tapia, O. L. White, and J. W. Lichtman, The interscutularismuscle connectome, PLoS Biol., vol. 7, no. 2, e1000032, 2009.

Jerry Chao received the B.S. and M.S. degrees incomputer science from the University of Texas atDallas, Richardson, in 2000 and 2002, respectively,from where he is currently working toward the Ph.D.degree in the Department of Electrical Engineering.

From 2003 to 2005, he was involved in the devel-opment of software for microscopy image acquisitionand analysis at the University of Texas SouthwesternMedical Center, Dallas. He is currently a ResearchAssistant in the Department of Electrical Engineer-ing, University of Texas at Dallas. His research in-

terests include image and signal processing for cellular microscopy and thedevelopment of software for bioengineering applications.

Mr. Chao is a student member of the Biophysical Society.

E. Sally Ward received the Ph.D. degree from theDepartment of Biochemistry, Cambridge University,Cambridge, U.K., in 1985.

From 1985 to 1987, she was a Research Fellowat Gonville and Caius College, while working at theDepartment of Biochemistry, Cambridge University.From 1988 to 1990, she was Stanley Elmore SeniorResearch Fellow at Sidney Sussex College, whereshe was involved in research at the MRC Laboratoryof Molecular Biology, Cambridge, U.K. In 1990, shejoined the University of Texas Southwestern Medical

Center, Dallas, as an Assistant Professor. Since 2002, she has been a Professorin the Department of Immunology, University of Texas Southwestern MedicalCenter, where she is currently Paul and Betty Meek-FINA Professor in molecularimmunology. Her research interests include antibody engineering, molecularmechanisms that lead to autoimmune disease, questions related to the in vivodynamics of antibodies, and the use of microscopy techniques for the study ofantibody trafficking in cells.

Raimund J. Ober (S87M87SM95) received thePh.D. degree in engineering from Cambridge Univer-sity, Cambridge, U.K., in 1987.

From 1987 to 1990, he was a Research Fellowat Girton College and the Engineering Department,Cambridge University. In 1990, he joined the Uni-versity of Texas at Dallas, Richardson, where he iscurrently a Professor in the Department of ElectricalEngineering. He is also an Adjunct Professor at theUniversity of Texas Southwestern Medical Center,Dallas. He is an Associate Editor of Multidimensional

Systems and Signal Processing and Mathematics of Control, Signals, and Sys-tems. His research interests include the development of microscopy techniquesfor cellular investigations, in particular at the single molecule level, the study ofcellular trafficking pathways using microscopy investigations, and signal/imageprocessing of bioengineering data.

Dr. Ober was an Associate Editor of IEEE TRANSACTIONS ON CIRCUITS ANDSYSTEMS and Systems and Control Letters.

Software Framework for Analysis of Complex Microscopy Image Data

Documents

Transcript of Software Framework for Analysis of Complex Microscopy Image Data