Tracking Cell Signals in Fluorescent Images

Victoria Gor Tigran BacarianJet Propulsion Laboratory, Institute for Genomics and Bioinformatics,Instruments and Science Data Systems and Department of Computer Science,victoria.gor@jpl.nasa.gov University of California, Irvine

tbacaria@uci.edu

Michael Elowitz Eric MjolsnessDepartment of Biology and Applied Physics, Institute for Genomics and Bioinformatics,Caltech and Department of Computer Science,melowitz@caltech.edu University of California, Irvine

emj@uci.edu.

Abstract

In this paper we present the techniques for tracking cellsignal in GFP (Green Fluorescent Protein) images ofgrowing cell colonies. We use such tracking for both dataextraction and dynamic modeling of intracellular processes.The techniques are based on optimization of energyfunctions, which simultaneously determines cellcorrespondences, while estimating the mapping functions. Inaddition to spatial mappings such as affine and Thin-PlateSpline mapping, the cell growth and cell division historiesmust be estimated as well. Different levels of jointoptimization are discussed.

The most unusual tracking feature addressed in this paperis the possibility of one-to-two correspondences caused bycell division. A novel extended softassign algorithm forsolutions of one-to-many correspondences is detailed in thispaper. The techniques are demonstrated on three sets ofdata: growing bacillus Subtillus and e-coli colonies and adeveloping plant shoot apical meristem. The techniques arecurrently used by biologists for data extraction andhypothesis formation.

1. Introduction

1.1 Motivation

The deployment of in vivo live confocal microscopy hasenabled scientists to capture gene expression data and beginto create computational models of developing cellularorganisms. Such systems include intracellular molecularregulation networks combined with intercellular signals andtransport, cell growth and proliferation, and mechanicalinteractions between cells, resulting in complex interaction

networks with the ability to control the development ofmulticellular organisms.

The amount and quality of collected expression data issignificant enough that scientists are able to hypothesizemany of the underlying control circuits, but the experimentaldata of important molecular players and interactions aremost often incomplete, and additional hypotheses areneeded to explain their spatial and dynamical behavior [1].Mathematical modeling provides a powerful method fordescribing and testing hypotheses about developmentalbiological systems. Not only can hypotheses be tested to seeif they account for the observed data but predictions can bemade for new experiments.

The data provided by the confocal imaging technique isavailable in the form of an image time series, quantificationof which is essential to creation of viable models. Differentimage processing algorithms are used to extract cellcompartments and GFP fluorescence intensities withinindividual cells [2,3]. It is usually assumed that the GFPintensity is linearly related to the amount of protein, andhence the average intensity within a cell is interpreted as arelative protein concentration.

Once the cell boundaries and the signal within cells isextracted from individual images, the task of finding thecorrespondence between cells at different time points, sothat temporal developmental signal can be extracted for eachcell, remains. This is the problem that we will address in thisarticle.

1.2 Background

In general, the problem of finding correspondences ormatches between image objects is a fundamental problem incomputer image analysis. In its worst case (when eachobject in one image can potentially correspond to every

object in another image) this problem may be NP-complete.In practice this means that in order to find optimalcorrespondence, combinatorially many matches have to beconsidered. For large data sets, such a search leads toprohibitive computational times. In addition, the problemcan be further complicated if unknown transformation(mapping) is applied to the objects in different imageframes. In this case in addition to finding optimalcorrespondences between points, the transformationsinfluencing object attributes (such as coordinates) have to beestimated as well. This is often the case in fluorescentimaging.

Many solutions to point matching and graph matching areavailable in the literature. The state of the art work is basedon joint estimation of correspondence and spatial mappingsvia optimization of energy function [5,6,7,8]. The generalframework for such optimization is proposed by Gold et al.[4]. This framework uses the methods of deterministicannealing [9,13] in conjunction with soft assign [10,14,15]and clocked objectives [11] to produce an optimizingnetwork and a corresponding Energy function.

The success of such approach is dependent on the designof energy function in conjunction with the choice ofoptimization technique. The typical energy function consistsof two parts E = Ep + Econs. Energy of constrains (Econs) istightly related to optimization method being used, andEnergy of the problem (Ep) is tightly related to the problem.Ideally Ep should use all the data and information available,to estimate the error of given correspondence.

Of crucial importance in designing Ep is accurate selectionof mapping function. If the image data has undergone affinetransformation, the natural choice for mapping function isobviously an affine transformation. If the parameters of suchtransformation are not known, they must be estimatedsimultaneously with correspondence. Another example ofmapping is Thin Plate Spline, proposed by Chui andRangarajan [12] for matching objects in brain MRI. It isobvious that the Ep part of energy function variessignificantly, depending on the data and it’s behavior.

2. Problem description and objectives

In this article, we will address tracking, matching andmodeling signals in fluorescent imagery, using the latteroptimizing network. More precisely, the problem is: Given asequence of images (movie) depicting cell arrays, determinethe positions of each cell at each time point (track cellsthrough time), while recognizing cell division events andrecording cell lineage. Cell death is not observed in theimage data under study, however during growth cells oftendisappear into the “out of focus” regions.

Cell centroids and other attributes have been previouslyextracted from such images using various image processingalgorithms as described briefly in Section 6. Extraction ofcell attributes from individual images is not a focus of thispaper. Instead, we concentrate on determiningcorrespondences between already extracted cells inconsecutive images. Determining such correspondences

between cells can be very challenging, especially for lowsampling rate, when cells move a lot from frame to frame. Itoften requires modeling of cell motion. In addition, thelimitations of imaging process and image processingalgorithms produce mistakes in extracted cell attributes,therefore making matching job even harder.

The main challenge here is to design the Ep and mappingsadequate for extraction and modeling of temporal cellsignals. The form of some of these mappings is known apriori. For example, it is usually known that the data mightundergo rotation and translation or other Euclidean or affinetransformations during development. But there are othermappings present in cell colonies that are less clear andextremely important. Such mappings are caused by cellgrowth and cell division (growth transformation), and theyare by far less known and they are actually the object of thescientific study. In fact, the growth transformation involves(is a function of) cell products and is tightly interconnectedwith the dynamic network of the cell colony under study. Inprinciple, as far as cell growth and cell divisiontransformations (mappings) are concerned, we areconfronted with a “chicken and egg problem”. To extract thesignal from the data, one must hypothesize the mapping, buttwo hypothesize the mapping one must extract the signalfrom the data. In this way our objectives form a closed loop(Figure 1). Each node in a loop enables next node. Onceagain a simultaneous solution is desired, but it is onlypossible if the form of growth and cell division mappingscan be hypothesized. Practically, this problem is solved byincremental improvement of Ep, and analysis of obtainedsolutions.

In such approach we would start with a very simple Ep,and we would apply it to the best obtainable data. Such datamust be collected at the fastest sampling rate possible, sothat cell displacement due to growth and cell divisiontransformations is negligible. That allows us to ignoregrowth transformation, while extracting a temporal cellsignal, and then to hypothesize such transformation givenextracted signal. Once the form of growth transformation ishypothesized, the simultaneous solution for growth mappingand correspondence can be obtained, and then theinteractions of these transformations with other celldata/parameters can be hypothesized and formulated as adynamic network.

Figure 1. The objectives loop. Each objective enablesthe next one.

Extract temporal cellsignals (cell data overtime)

Hypothesize and learncell growth and celldivision transformations

Determine interactions ofthese transformationswith other cell variables

Formulate plant as aDynamic network

3. Approach to solution: Sequential vs.Simultaneous solutions

Our general approach to the solution is to encode problemgoals as terms in an objective function and to determine thepoint (cell) correspondence and transformation parameters,which optimize the objective function. Such optimizationcan be done in steps with simpler energy functions, whereeach optimization step assumes the results from previousoptimization steps (Figure 2), or it can be done jointly withone more complicated (Eq. 3 and Eq. 5) energy functionmodeling all the unknown mappings.

Since after each step in sequential processing, the overallsearch space is significantly reduced, and since the energyfunctions for such processing are generally much simpler,the sequential solution is faster and easier to implement. Thedrawback, of course, is that not the entire search space hasbeen evaluated and the optimal solution with respect to allvariables might be less accurate. However, if the sequentialprocess is stated in such way that most reliable optimizationcomponents are done first, the accuracy of the results inpractice might even exceed the one obtained with jointoptimization. Often, for easier data sets, when thedisplacement of the cells in consecutive images is small thesequential approach outperforms joint optimization (SeeSection 6). Nevertheless, this is not the case with morechallenging data sets. In these cases, joint optimization tendsto outperform sequential processing.

Our approach to cell tracking in fluorescent imagery willstart with the simple objective functions suitable forsequential optimization (Eq.1) and (Eq.2), and it willprogress to the more complicated energy functions, suitablefor simultaneous optimization.

4. Sequential Solution

In sequential optimization the energy function is split intocomponents and each component is optimized separatelywithout feedback. Sequential optimization for the problemof extracting temporal cell signals from GFP data is depictedin Figure 2. This approach first determines point matchingin each pair of two consecutive images. It ignores cellgrowth and cell division mappings and calculatescorrespondences using Ep as in (Eq. 1).

Figure 2. Sequential solution for tracking cell signals.

The correspondence matrix

€

Mij includes slack variables,

thereby allowing non-matches for outlier points. The output

of point matching are matched pairs

€

xi,y j{ } where

€

xi is

the point (cell) i from previous image (or nothing) and

€

y j is

cell j from following image (or nothing). The overall energyfunction to be minimized is a total error (Euclidian distancebetween cell positions in consecutive images) of all matches

scaled by the expected variance of this error (

€

σ 2), and isgiven by (Eq. 1). The mapping is assumed to be affine (withparameters A) and is known for some of the data. When it isnot known, A is optimized jointly with correspondence [4].

€

Ep = Mij Axi − y j2σ 2( )

i, j=1∑ (Eq.1)

Because of cell divisions, which are frequent in ourmicroscope data, the previous image in the sequencecontains smaller number of cells then following image.Therefore, there usually will be a number of

€

yk that

matched to nothing in previous image. Such

€

yk are deducedwith high probability to be a product of cell division. Thenext step in sequential approach is to determine parents andsiblings for such

€

yk (given found matched pairs). Theobjective function for this process is given by Es in (Eq. 2).The first term is the Euclidian distance between two siblingsfrom the same image. It is scaled by expected variance of

this distance (

€

σ (1)2 ). The second term is the Euclidian

distance between

€

yk and the parent of it’s sibling

€

yl scaled

by expected variance of this distance (

€

σ (2)2 ). The objective

function is defined to minimize total error between currentcells and their siblings and between current cells andsibling’s parents. The correspondence matrix entry

€

Lkl will

optimize to 1, if cells

€

yk and

€

yl are siblings with commonparent in the previous image, and it will optimize to 0otherwise. Just like

€

Mij ,

€

Lkl includes slack row and

column for possible non-matches. Even though in our datasets cells are not expected to dye or appear out of nowhere(orphans), this often happens in real imagery due to imaginglimitations. For future reference, we always include slackrow and column in all correspondence matrices mentionedin the rest of this article.

€

Es = Lklyk − yl

2σ (1)2 +

yk − previous(yl )2σ (2)2

k,l∑ (Eq.2)

The remaining component of the sequential solution istracking. In general, tracking combines pairwise matches toproduce the tracks (paths) of the cells through the entireimage sequence (movie) and builds the lineage tree. If thecells move randomly in the image and do not follow anytrajectory that can be modeled, or if such trajectory is notimmediately observable (as is often the case with sequentialapproach), not much can be gained from tracking in terms ofaccuracy of the results. In this case, pairwise matches aregiven a priori, and tracking is just a greedy searchimplemented via simple “bookkeeping” algorithm. The onlyimprovement this algorithm makes in terms of accuracy issome inference of missing or merged cells, resulted from the

Tracking ModelFitting

Pointmatching

Siblingmatching

€

xi = previous(y j )

€

yl = sibling(yk )€

xi,y j{ }

€

yl ,yk{ }

€

trackdata

mistakes of cell extraction algorithms. The result of thetracking algorithm is a list of tracks (track data), each trackspecifying the coordinates and attributes of one cell in allimages it exists in. Once such tracks are extracted and theattributes of cells over time can be quantitatively examined,the model for cell motion and growth can be assumed andfitted into existing data (Model Fitting step).

5. Simultaneous solutions

5.1 Pairwise matching with cell division

In simultaneous solution the components of sequentialsolution are combined in joint optimization. We first attemptto perform joint optimization of cell matching and siblingmatching. We use Robust Point Matching algorithmutilizing Thin Plate Spline (TPS) transformations withsoftassign embedded in deterministic annealing loop [12].The full objective function ( E = Ep + Econst) in suchmatching problem is given in (Eq.3)

€

E = Mi,α, j θ −| xi,α −ϖ(y j ) |2][

σ 2

i,α , j

∑

−1β

Mi,α , j (log Mi,α, j )∑

−λ1Tr(ωT ˆ Φ ω) − λ2Tr[(d − ˆ I )T (d − ˆ I )]

− µk Mi,α, jr(k )∑

k∑

(Eq.3)

The ! !

€

Mi,α , j are several matrices with

€

α != ! !{ !1 !} !, !

€

α !=!{!1!,!2!}!,!!o !r !

€

α ! ! != ! !{1 !, !2 !, 3}, ! obeying ! !d !o !u !b !l !y ! !stochastic ! !r !u !l !e !s ! !r !( !k !) ! asdescribed in Section 5.3. The

€

ϖ(y j ) is thin plate spline

transformation of the vector

€

y j as shown in (Eq. 4)

€

ϖ(y j ) = y jd + φ(y j ,yk )ωkk∑ (Eq.4)

!He !r !e ! !affine ! !matrices !

€

d ! !a !n !d ! !warping !v !e !c !t !o !r !s !

€

ωk ! !constitute!t !h !e ! !T !P !S ! !coefficients !, ! !a !n !d !

€

φ ! !is ! !t !h !e ! !corresponding !k !e !r !n !e !l !

!function.

€

ˆ Φ is composed of

€

φ(y j,yk )values of the kernel

function. The first term of this function is the Euclidiandistance error of hypothesized correspondence. Parameter

€

θregulates the probability of cell matching to nothing.Increasing

€

θ decreases the probability of non-matchesduring stochastic optimization and vise versa. The third termin E is the standard thin-plate spline regularization termwhich penalizes the local warping coefficients

€

ω . ! The forthterm constraints affine mapping d by penalizing the residualpart of d which is different from an identity matrix I.

€

λ2 ! !a !n !d !

€

λ1 ! !coefficients ! !penalize ! !t !h !e ! !affine ! !a !n !d ! !warping ! !p !a !r !t !s ! !o !f ! !t !h !e !

!T !P !S ! !accordingly !. !Normally !

€

λ2 ! !is set ! !small ! !t !o ! !a !d !j !u !s !t ! !the !!affine ! !coefficients ! !d ! !b !e !f !o !r !e ! !w !. The second and fifth termsof the energy function represent the constraints imposed onthe problem (Econst). The second term is an entropy barrierfunction with the temperature parameter T=

€

1 β . It is usedin deterministic annealing step [9]. The fifth term is astochastic optimization term with the Lagrange parameters

€

µk corresponding to the rules r(k) realized by softassign[10]. The details of novel normalization rules for thispairwise matching are presented in Section 5.3.

5.2 Tracking with cell division and affinetransformation

Finally, we attempt to combine all steps of sequentialprocess in one joint optimization. This is achieved with theEnergy function in (Eq. 5).

€

E = e−λtt∑

Lα ,it xi

t − yαt 2

2σ (1)2 −θ[ ]

i,α=1∑

+ yαt+1 − At • yα

t 22σ (2)

2

α=1∑

+ Nβαt yβ

t+1 − At • yαt 2

2σ 2 −θ [ ]α ,β =1∑

+ e− ′ λ t

t∑

µit Lαi

t −1α= 0∑

i=1∑ + ηα

t Lαit −1

i=1∑

α=1∑

+T Lαit logLαi

t −1( )α ,i= 0∑

+ e− ′ λ t

t∑

ˆ µ αt Nβα

t −1β = 0∑

α=1∑ + ˆ η β

t Nβαt −1

α= 0∑

β =1∑

+T Nβαt logNβα

t −1( )α ,β = 0∑

(Eq.5)

This energy function is minimized by estimating track

coordinates at time t (

€

yαt ) and affine transformation

€

At ,while minimizing: total error between estimated tracks and

true point coordinates at time t (

€

xit ) (first term of first sum);

total error between consecutive points of each track, givenestimated affine (second term of first sum); total error

between sibling track (

€

yβt ) and it’s transformed parent

(third term of first sum). The remaining two sums of thisenergy function constitute Econs (the energy of constrains).As before, there is an entropy barrier function terms (lastterms of second and third sums) with annealing temperatureT; and unique correspondence optimization terms with

Lagrange parameters

€

µit and

€

ηαt for the correspondence

between tracks and points at time t (

€

Lαit ), and with

Lagrange parameters

€

ˆ µ αt and

€

ˆ η βt for the correspondence

between siblings at time t (

€

Nβαt ). Note that this energy

function has several correspondence problems to solve

simultaneously:

€

Lαit and

€

Nβαt for every time point t. Solving

for L and N performs joint optimization for track and siblingmatching, and optimizing these matrices jointly for everytime point t performs joint optimization of tracks. Weincluded entropy and unique correspondence constraints foreach correspondence problem and scaled the energy

function with

€

e−λt , thus placing more weight on earliermatches to insure forward directionality of the solutiondynamics.

As was stated in the background section the jointoptimization is made possible by softassign, deterministicannealing and clocked objectives methods. The softassign isbased on Sinkhorn’s theorem [16], which states that adoubly stochastic matrix is obtained from any positivesquare matrix by alternating row and columnnormalizations. Such a normalization process is directlyrelated to solving for Lagrange multiplier parameters. Theoriginal entries of the stochastic matrix (in this approach L,and N) are typically an error term

€

Qijobtained by setting thepartial derivatives of Energy function E with respect tocorrespondence matrices to zero (

€

∂E ∂Lαit = 0 and

€

∂E N βαt = 0 ). If original

€

Qij> 0, the softassign insures that

€

Qij ≈1j∑ for all i and

€

Qij ≈1i∑ for all j. Softassign in

conjunction with deterministic annealing can find globalminimum in the assignment problem [9]. As the temperatureis reduced the doubly stochastic matrix approaches apermutation matrix imposing an additional constraint

€

Qij ≈01

.

To obtain closed form solutions for unknown parametersthe method of Clocked Objectives is used. In this methodthe unknown parameters are obtained by differentiatingenergy function with respect to these parameters and settingresult to 0 (to find the minimum). The closed-form solutionsare obtained in the iterative scheme [11]. The iterativescheme for the given problem is represented by formula in(Eq.6). This formula states that in iterative scheme first thetrack values

€

yαt have to be estimated by calculating an

analytical solution to

€

∂E ∂yαt = 0 , then assuming found

track values; the transformation matrix

€

At has to beestimated by calculating an analytical solution to

€

dE ∂At = 0 . Then assuming the current estimates fortrack values and affine transformation, the correspondencematrices

€

Lαit and

€

Nβαt are determined by solving for

Lagrange multipliers

€

µ,η, ˆ µ , ˆ η via previously describedsoftassign technique.

€

F⊕ = F y,A, (L,µ),(L,η , (N, ˆ µ ),(N, ˆ η (Eq.6)

5.3 Normalization rules for pairwise matching

In order to facilitate Robust Point Matching Thin-PlateSpline (RPM-TPS) algorithm [12] for more than onepossible mapping several correspondence matrices havebeen used (indexed by

€

α ). One matrix representing a usualone-to-one match, and the other two, similarly, representingpossible split into a one-to-two match (See Figure 3). Theexclusion requirement has been implemented throughmodified row and column unique correspondenceconstraints.

Figure 3. Paired stochasticity of corresponding joinedrows.

€

s0i ,

€

s1i , and

€

s2i are corresponding summations for the

three matrices.

As before, a unique match constraint along columnsinsures that there is no more than one match for each pointfrom the second set. Therefore, the normalization ruleasserting unique j point for each i point is the same (Eq. 7).

€

Mi,α , ji,α∑ =1 (Eq.7)

However, for j-summations a different rule is in effect.There can be either parent-to-one or parent-to-two daughtercorrespondences, but not both. This leads to(Eq.8)

€

Mi,α , j α=1j∑ = Mi,α, j α= 2

j∑ =1− Mi,α , j α= 0

j∑ (Eq.8)

If corresponding summations for the three matrices

(including the slack elements) are:

€

s0i ,

€

s1i , and

€

s2i , then the

above rules are satisfied by multiplying each I-raw by

€

κ0i ,κ1

i, and

€

κ2i accordingly as in (Eq.9).

Paired constraint of corresponding joined rows from thefirst and the second matrices as well as the first and the third(Figure 3), insure that annealing procedure will leave eitherone match for a point from the first set in the upper matrices,or two possible matches on the same-indexed rows in thematrices below, but never both of them at the same time.Because each member of each matrix multiplies a separateterm in the objective function (see Eq. 3), this approachprovides both simultaneous solution of the split-matchingproblem and flexibility in coupling different points fromdifferent matching possibilities.

1

1)2(10

=

=+j

ii

s

ssis1

js

is2

is0

€

k0i =

2si +1/2( )

2

k1i =

s2i

si +1/2

k2i =

s1i

si +1/2,

where

si =14

+ s0i / s1

i + s2i( )

(Eq.9)

6. Experiments

We have used three data sets in our analysis: 3-dimensional images of a growing plant shoot-apicalmeristem, 2-dimensional images of developing colonies ofbacillus Subtillus, and 2-dimensional images of developingcolonies of e-coli. The extraction of cell coordinates from 3-d images has been performed via standard gradient descentmethod by the team of scientists from Lund University,Sweden, under the leadership of Henrik Jönsson. Theextraction of cell coordinates and other cell attributes from2-d images have been previously accomplished by theCaltech team under the leadership of Michael Elowitz. Theyused iterative erosion/dilation algorithms for this purpose.

The e-coli and bacillus Subtillus data sets are 2-dimensional. It is possible to visualize and ground truththese data (Figure 4, Figure 5). The bacillus Subtillus and e-coli are not rigidly attached to each other and can movenoticeable distances from frame to frame. Such motion isthe main challenge of these data sets.

The shoot apical meristem data is 3-dimensional. In thisdata, cell walls are attached to each other, and interactmechanically preventing large displacements. In this datacells appear to move outwards (Figure 6b)), but such motioncan be observed only over large time lapses (about 10 timepoints); locally cell displacements appear random. This canbe modeled as an additional track constraint in the jointtrack optimization method. It is more difficult to visualizeand ground-truth this data. To evaluate roughly theperformance of this data track statistics and visualizationtools have been used.

Currently, tracking (matching all pairs of imagessimultaneously as in Eq. 5) suffers from track fragmentation(breaking up of one cell track into few segments). The mainreason for this is that number of tracks has to besignificantly larger then number of cells in individual image(especially in the beginning of the sequences). Therefore,for the goodness of match based on Euclidian distance, it isless costly to fit few tracks into set of points, rather thenone. We attempted to control this problem with slackparameter

€

θ and by adding of an additional term (thesquared sum of correspondence matrix entries, excludingslacks), but did not get much more control over the problem.

This study still continues. Therefore for comparison ofsequential and simultaneous methods, we focus onsequential approach in Eq. 1) and Eq. 2) and jointoptimization approach in Eq. 3).

In bacillus Subtillus data, estimating track values viasequential approach slightly outperformed the jointoptimization approach. In joint optimization approach 68points were matched wrong out of total 2217 pointscollected from 22 images, therefore amounting to 3% error.In sequential optimization (pairwise matching with TPS)approach 40 points were mismatched, therefore amountingto 1.8% error.

Figure 4 depicts the tracking of one colony (out of 6) on22 consecutive images (only 4 early consecutive images aredisplayed here due to the lack of space). The correspondingcells are numbered with the same track number. The trackidentification numbers of found siblings are displayed inparenthesis under current track number.

In e-coli data (Figures 5b), 5c)), the joint optimizationapproach overall outperforms sequential approach,producing 5.35% error vrs. 6.44% error (in sequentialapproach). The e-coli data is different from bacillusSubtillus data, it is complicated by the presence of longirregular cells as in Figures 6b) and 6c). The energyfunctions were mainly designed for Subtillus data, withoutconsideration for long irregular cells, therefore the worstperformance has been expected. However, the jointoptimization method demonstrated to be more robust byoutperforming sequential method on this difficult data.Figure 6a) depicts the percentage error for 35 pairwisematches (36 individual images). Since the total number ofcells in each image of the sequence is different there is nodirect relationship between overall error and error/perimage. For example, if some early image contains two cells,one of which is matched wrong, there will be 50% error forthis image, but overall percent error increase will benegligible. The results of sequential optimization aredisplayed with dashed line, and the results of jointoptimization are depicted with solid line. Note that bothalgorithms resulted with 0 error for first 25 images, howeverthe error rates grow with complexity (number of cells) andan increase in missed (by segmentation process) cells.

Another interesting factor to observe is that the errorsproduced by sequential and joint optimization approachesare somewhat orthogonal, meaning that the erroneousmatches seem to be different for both approaches. Insequential approach most of the errors were produced inmatching of the siblings: 11.9% error for sibling matching,4.2% error for same cell matching. In joint optimizationapproach 4.2% error was accomplished in sibling matching,and 5.8% error was produced in same cell matching.Moreover, joint optimization errors appear to be of globalcharacter. For example, note erroneous track 78, suchmismatch is easily detected with a sanity test based ondistance only. Sequential optimization errors are mainlylocal. Note erroneous tracks 3 and 54. This leads to thepossibility that overall results can be improved bycombining the results of both algorithms.

Finally, we present some preliminary results of trackingsignal in shoot apical meristem data. In the absence ofground-truth, we have used statistics to select reasonablesolutions. Such statistics include the average density oftracks and the average standard deviation of tracks, whichare inversely related; and also include the number oforphans (newly appearing cells without matched parent) andnumber of deaths (disappeared cells). The latter twomeasures ideally should be very small, but such criterioncannot be enforced rigidly. Extracting these statistics underpresent conditions assists in selecting reasonable solutions,but is not sufficient for comparative analysis.

One such solution is depicted in Figure 5. For reasons oflegibility only few tracks are included in the plot of Figure5-a). The star denotes the beginning of the track, andbranches represent cell divisions. The newly born cells areconnected to their parents with black dashed lines. One canobserve that cell motion appears locally random, but aglobal outward growth tendency is observed as well. Aclearer view of an outward growth tendency form aninterpolated total displacement field for 10 time points, isdepicted in Figure 5-b).

Figure 4. Pairwise matching for tracking bacillusSubtillus data.

a) Percentage error for 35 pairwise matches producedby sequential optimization approach (dashed line) andjoint optimization approach (solid line).

b) Matching cells in e-coli images 31/32 withsequential optimization approach. Matches displayedon raw image data.

c) Matching cells in e-coli images 34/35 with jointoptimization approach. Matches displayed onsegmented image data.

Figure 5. Sequential vrs. joint optimization in e-colidata.

6. Conclusions and plans

In this paper we have demonstrated sequential and jointoptimization techniques applied to tracking cell signals inGFP images. There are two main challenges that suchtracking has to confront. The first challenge is matchingallowing one-to-two correspondences. We have addressedthis challenge with an extended softassign algorithm, whichperforms well and presents a valuable general tool for thesolution of one-to-many correspondences. The secondchallenge is missing cells mistakes produced by cell-segmentation. Some of these mistakes were addressedsuccessfully via track inference in the sequential approach.The more general approach, which performs joint estimation

of track values, can handle missing cells better, but it suffersfrom track fragmentations. Such an approach is moreimportant when cell motion has structure or can beexpressed as a function of cell variables. Therefore, it ismore likely to be used in the final dynamic model. In thefuture, we plan to deal with cell segmentation (extraction)mistakes by performing cell segmentation and cell matchingjointly. This is especially useful since there is no automaticway to assess the goodness of the segmentation, but there isa way to assess goodness of the match (number ofunmatched points).

a) Selected tracks generated for shoot apical meristemdata.

b) The interpolated displacement field for large timestep.

Figure 6. Tracking Plant Shoot Apical Meristem Data.

Our current software is used by biologists to extract thecell signals from GFP imagery. Given such signals,scientists are able to hypothesize the details of underlyingcell processes, to model these processes in dynamicnetworks and to make predictions about organism behavior.

On the other hand, such models and predictions allow forbetter signal extraction (tracking algorithms), producingmore accurate data for use in biological research.

Acknowledgements

This work was supported by NSF:FIBR award numberEF-0330786. We are grateful to Marcus Heisler, Elliot M.Meyerowitz, Henrik Jönsson, and Joe Roden for their rolesin providing test data.

References

[1] !H!e!n!r!i!k! ! Jönsson!, M!a!r!c!u!s! !H!e!i!s!l!e!r!, !G!.! !V!e!n!u!g!o!p!a!l!a! !R!e!d!d!y,! V!i!k!a!s!!A!g!r!a!w!a!l, !V!i!c!t!o!r!i!a! !G!o!r, !B!r!u!c!e! !E!.! !S!h!a!p!i!r!o!,! !E!r!i!c! !M!j!o!l!s!n!e!s!s!!,! !E!l!l!i!o!t! !M!.!!M!e!y!e!r!o!w!i!t!z, “Modeling the organization of the WUSCHELexpression domain in the shoot apical meristem”, Bioinformatics,vol.00, no.00, 2005.[2] Luc Vincent and Pierre Soille, “Watersheds in digital spaces:an efficient algorithm based on immersion simulations”, IEEETrans. Patt. Anal. Mach. Intell., vol.13, no.6, June 1991.[3] Pierre Soille, Edmond J. Breen, Ronald Jones, “Recursiveimplementation of erosions and dilations along discrete lines atarbitrary angles, IEEE Trans. Patt. Anal. Mach. Intell., vol.18,no.5, may 1996.[4] Steven Gold, Chien- Ping Lu, Anand Rangarajan, SugunaPappu, and Eric Mjolsness,”New algorithms for 2-D and 3-D pointmatching: pose estimation and correspondence”, PatternRecognition, 1998.[5] S. Gold and A. Rangarajan, “A graduated assignment algorithmfor graph matching”, IEEE Trans. Patt. Anal. Mach. Intell.,1996.

[6] A. D. J. Cross and E. R. Hancock, “Graph matching with adual-step EM algorithm.”, IEEE Trans. Patt. Anal. Mach.Intell.,1998.[7] S. Sclaroff and A.P. Pentland, “Model matching forcorrespondence and recognition”, IEEE Trans. Patt. Anal. Mach.Intell.,1995.[8] Anand Rangarajan, Haili Chui, Eric Mjolsness, Suguna Pappu,Lila Davachi, Patricia S. Goldman-Rakic, and James S. Duncan,“A robust point matching algorithm for autoradiograph alignment”,Medical Image Analysis, 1997.[9] J.J. Koslowsky and A.L. Yuille, “The invisible hand algorithm:Solving the assignment problem with statistical physics”, NeuralNetworks, 1994.[10] S. Gold and A. Rangarajan, “Softmax to Softassign: Neuralnetwork algorithms for combinatorial optimization”, Journal ofArtificial Neural Networks, 1996.[11] E.Mjolsnessand W. Miranker, “A Lagrangian Formulation ofNeural Networks II: Clocked Objective Functions andApplications”, Neural, Parallel and Scientific Computations, vol 6no. 3, pp. 337-372, 1998.[12] H. Chui and A. Rangarajan, “A new algorithm for Non-RigidPoint Matching”, CVPR, 2000.[13] A.L. Yuille, J.J Koslowsky, “Statistical physics algorithmsthat converge”, Neural Computation, 1994.[14] D.E. Van den Bout, T.K. Miller, “Improving the performanceof the Hopfield-Tank neural network through normalization andannealing”, Biological Cybernetics, 1989.[15] P.D. Simic, “Statistical mechanics as the underlying theory of‘elastic’ and ‘neural’ optimizations”, Network, 1990.[16] R.Sinkhorn, “A relationship between arbitrary positivematrices and doubly stochastic matrices”, Annals of MathematicalStatistics, 1964.