Marker-Based Tracking of Subsurface Locomotion in Noisy Environments

Miguel M. Serrano, Sarah S. Sharpe, Daniel I. Goldman, Patricio A. VelaGeorgia Institute of Technology{mserrano6, pvela}@gatech.edu

Abstract

This work presents an automated, marker-basedtracking system for monitoring the subsurface locomo-tion strategies of burrowing animals, in particular smallreptiles. High speed x-ray imaging was used to visual-ize movement of a subject within a controlled granularenvironment. Given the images, the system returns thetrajectories of markers placed on the subject’s body andlimbs. Unlike existing methods that require unique cor-respondences between markers in subsequent images,the proposed method is designed to handle false positivemarker detections arising from the noise inherent to x-ray imaging. A Bayesian approach to trajectory estima-tion using linear models with Gaussian noise leads to acombined Kuhn-Munkres algorithm and Kalman filter-ing formulation. This is the first step towards a robust,marker-based tracking system for articulated subjectsundergoing locomotion in noisy environments.

1. Introduction

Recent efforts studying reptiles like the sand-divingand swimming sandfish [2, 9] seek to understand howthe interaction between the animal and its environmentproduce subsurface locomotion. Identifying the inter-action mechanisms for modeling purposes requires ex-tensive investigation [10]. Accurate model estimation iscontingent on the ability to extract the motion trajecto-ries of the animal’s body while fully submerged in themedia. The difficulty in studying the subsurface loco-motion of these animals arises from the opaque granularmedia, which renders traditional visible light imagingacquisition techniques useless.

Subsurface imaging is aided by the use of high speedx-ray technology [3, 6]. Through the strategic place-ment of high density markers on the subject of inter-est, trajectories of certain body landmarks can be ex-tracted. To be of practical utility however, automatedmethods for trajectory extraction are necessary. Exam-

ples of automated marker-based tracking algorithms forx-ray imaging include [1, 4]. The X-Ray Reconstruc-tion of moving Morphology (XROMM) algorithm hasbeen sucessfully used for motion analysis [1, 5]. It isa template-based approach that requires one-to-one cor-respondences between detected markers across frames.The technique in [4] uses active appearance models tolearn a low-dimensional representation of the subject’sshape. Such representations require a priori trainingand a kinematic model that truly has a low-dimensionalparametric representation. In the present study, imagingnoise associated with the x-ray process produces a one-to-many marker correspondence problem, and the rep-tile motion has many degrees of freedom, meaning thatthese approaches are not suitable for the present needs.

This paper presents an automated computer visionsystem for robustly tracking landmarks placed on an ar-ticulated, burrowing animal. The system uses blob de-tection to identify candidate markers, and assumes a 2Dmotion model for the markers. A recursive Bayesianestimation strategy for tracking leads to a combinedKuhn-Munkres algorithm and Kalman filtering methodfor estimating the marker trajectories. The combinedapproach results in a correspondence cost function thatincorporates the filtered states into the costs.

2 Image Acquisition

Animals moved in a container filled with approxi-matley spherical glass bead (diameter, d ≈ 0.3mm or3mm, and density, ρ = 2.5 g

cm3 ), which was placed be-tween the x-ray source and image intensifier to producetop view (or dorsal) images of animal burial. The mediawas approximately 10 cm deep. Further information onthe methods and techniques can be found in [9].

X-ray recordings of the animals observed entry andmovement through the granular medium. Figure 1shows an extracted image with a sandfish submergedin coarse media. The contrast between the animal’sbody and the environment is too low to admit trackingof the body. Thus lead markers (1 mm in diameter) are

(a) Mask boundary

1 cm

(b) Detections

Figure 1. Masking the marker detectionsfor shovel-nosed snake.

placed along the spine of the animal (and also on thelegs if appropriate) to enhance the contrast and facilitatetracking. As seen in Figure 1, these markers stand outas dark regions in contrast to the surrounding environ-ment. Under ideal circumstances the markers maintainsufficient contrast for tracking of subsurface locomotionthroughout the image acquisition. However, in less thanideal circumstances there are important impediments totracking. Digital radiography has quantum noise andsensor noise. Further, the coarseness of the media im-pacts the imaging contrast and texture, while the barriergenerates occlusions. In these cases, markers can dis-appear and some image regions can have marker-likevisual characteristics.

3 Marker Tracking

The proposed tracking solution is decomposed intotwo phases, the detection phase and the associationphase. By design the association phase also incorpo-rates filtering of the estimated marker trajectories.

3.1 Marker Detection

The lead markers on the subject have a roughly cir-cular appearance in the x-ray images. These markersare detected using a blob detection algorithm [8] rely-ing on high responses from a circular filter convolvedwith the x-ray image I . The circular filter is the Lapla-cian of Gaussian (LoG) filter,

r(x, y) = ∆Gσ ∗ I. (1)

where ∆ is the Laplacian operator and Gσ is the Gaus-sian kernel with standard deviation σ (also known as thescale of the filter). The highest response r for a markeris produced at the scale best matching the marker radius.

Thus, given an initial frame with manually taggedmarkers, unique blob detection kernels are designed todetect the markers of the subject, leading to a filterbank of LoG filters. The output of the convolutions arethresholded and the centroid of every disjoint detectionregion is classified as a candidate marker. As mentioned

(a) Mask boundary (b) Detections

Figure 2. Masking the marker detectionsfor shovel-nosed snake.

earlier, this approach also leads to false positives in thedetection process, and can also lead to missed detec-tions when the contrast between the surrounding mediaand the marker drops significantly. To reduce the quan-tity of false positives, a target mask region is utilized.Given the current estimate of the subject’s markers, amask is generated by considering only those image co-ordinates that are within a specified distance to any onemarker. Figure 2(a) shows the image with the maskregion, while Figure 2(b) shows all marker detections.The magenta square detections lie outside of the maskand are rejected, while the red circular detections lyingwithin the region are accepted for further processing.The blue crosses are the true marker locations.

3.2 Correspondence Matching

To successfully generate the frame-to-frame markerassociations, a correspondence matching algorithm isneeded. Following a Bayesian approach [?], the corre-spondence can be found by maximizing the probabilityassociated with all possible candidate correspondencesgiven the past detections and the current detection. As-suming a Markov process, the probability optimizationis reducible to a recursive estimation procedure. Letthe n marker states at time k be denoted by xi(k) fori ∈ S := {1, ..., n} which form the set X (k), and like-wise the m detected candidate marker positions be de-noted by yj(k) for j ∈ D := {1, ...,m} which formthe set Y(k). Assuming the markers are independentlyidentically distributed, the probability to maximize is

arg maxα(k,·),X (k)

n∏i=1

P (yα(k,i)(k)|xi(k))P (xi(k)|xi(k − 1)),

(2)where α(k, ·) : S → D is the marker to detected markercorrespondence function at time k. The optimization isover the correspondence function and the marker states.

Kalman Filter Model. For simplicity, first considerthe case of an individual single marker, labeled by i,with the correct correspondence. Assume that the dy-namics of each marker are linear time-invariant with

Gaussian uncertainty, plus that the (correspendence)measurement has Gaussian uncertainy,

xi(k) = Axi(k − 1) + wi(k),

yi(k) = Cxi(k) + vi(k),(3)

Both the state equation and the observation equationare affected by white stationary noise defined by wi ∼N (0, Q) and vi ∼ N (0, R), respectively. The Bayesianprocedure of equation (2) leads to a Kalman filter [?]for the marker trajectory for these given state equations.The Kalman filter first involves a prediction step,

x−i (k) = Axi(k − 1)

P−i (k) = APi(k − 1)AT +Q(4)

where x−i (k) and P−i (k) are the predicted state and co-variance obtained by propogating the previous estimatethrough the uncertain dynamics of (3). Then, given cur-rent measurement yi(k), there is a correction step,

xi(k) = x−i (k) +K(yi(k)− Cx−i (k))

Pi(k) = (1−KC)P−i (k)(5)

where the Kalman gain K is defined as:

K = P−i (k)CT (CP−i (k)CT +R)−1. (6)

Given that the distributions are all Gaussian, theKalman filter process maximizes a probability, as per(2), that is Gaussian. In this case maximization of theprobability is equivalent to minimization of the negativelog of the probability, as given by the cost function

ci(k) = ||yi(k)− Cx−i (k)||2W (k,i), (7)

where W (k, i) = (KT(P−i (k)

)−1K + (1 −

CK)TR−1(1− CK))−1, and ||x||2M = (xTM−1x).

Filtered Marker Correspondences. Suppose that allmarker states i ∈ {1, ..., n} are to be considered as wellas all currently detected markers j ∈ {1, ...,m}. TheBayesian filtering problem of equation (2) takes a morecomplex form, since n states should be estimated fromm possible associations. Optimization now requiresfinding both the associations and the filtered states. Us-ing (7), a candidate correspondence between marker iand detected marker j has the cost

ci,j = ||yj(k)− Cx−i (k)||2W (k,i). (8)

The association function α must reflect the set ofone-to-one correspondences maximizing (2), or equiva-lently minimizing the sum of (8) over all i ∈ S where

j = α(i), e.g.,∑i∈S ci,α(i). The Kuhn-Munkres al-

gorithm provides an efficient approach to identifyingthe optimal solution to the association problem [11]. Itsolves the dual problem of ensuring optimal assignmentof the maximum number of correspondence matcheswith the minimum effective net cost for the assignmentsmade. The algorithm structures the problem using acost matrix C := [ ci,j ], where ci,j is the cost of as-signing marker i ∈ S to candidate marker j ∈ D. TheKuhn-Munkres algorithm is an established solution tothe assignment problem given a feasible cost matrix.

4 Experiments

The algorithm was run on three different reptiles,the sandfish (Fig. 3), the shovel-nosed snake (Fig. 4),and the ocellated skink (Fig. 5), Nevertheless, the setuppresented can be used for the study of any small, ar-ticulated burrowing animals. Due to space considera-tions, one experimental results per animal will be given,with the selection being a more challenging sequencefor each animal. Given that the closest methods [1, 4]are not suited to track these sequences, comparison ismade with an approach that first generates correspon-dences then filters the results in order to show the im-portance of coupling the correspondence and filteringmore tightly. Each sequence was manually initializedon an initial frame where every marker was visible, anda second-order, constant velocity model was used forthe Kalman filter. Ground truth was obtained by manu-ally annotating 20 evenly space frames per sequence.

The tracking results, Figs. 3-5, refer to the proposedmethod as Filtered Association (FA) and the compari-son method as Associate-Then-Filter (ATF). Subfigure(a) shows a sample x-ray image and marker positionsfor each animal (Figs. 3-5) and subfigure (b) shows the

2 cm

(a) Sample frame

ATFATF TrajectoryFAFA Trajectory

(b) Tracking

0 2 4 6 8 10 12 1410

−2

100

102

Marker Id

Err

or: p

ixel

s

ATFFA

(c) Root mean squared error per marker

Figure 3. Sandfish tracking.

1 cm

(a) Sample frame

ATFATF TrajectoryFAFA Trajectory

(b) Tracking

0 2 4 6 8 10 12 14 1610

−2

100

102

Marker Id

Err

or: p

ixel

s

ATFFA

(c) Root mean squared error per marker

Figure 4. Shovel-nosed snake tracking.

trajectory for select markers (head, tail, and one limb ifapplicable). Subfigure (c) plots the root mean square er-ror plus the maximum and minimum errors per markerfor both FA and ATF. The importance of coupling theoverall procedure more tightly can be seen in Figure5(d), which plots the correspondence correctness forMarker 10 versus time. The top graph of Figure 5(d)shows that the ATF method generates incorrect corre-spondences that cannot be corrected over time while the(coupled) FA method does not lead to long-term incor-rect correspondences. The large error for Marker 12 inFigure 3(c) is due to long-term marker disappearance.Subfigures 3(c)-5(c) show that the proposed approachis accurate to a pixel when the marker is visible.

5 Conclusion

This paper presented an algorithm for automatedmarker-based tracking of articulated animals in noisyenvironments. The algorithm admits tracking of bur-rowing animals imaged using high speed x-ray. ABayesian-based optimal estimation formulation pro-vides for robust correspondences given false marker de-tections. The method successfully tracked three articu-lated reptiles: the sandfish, the ocellated skink, and theshovel-nosed snake.

References

[1] E. Brainerd, D. Baier, S. Gatesy, T. Hedrick, K. Met-zger, S. Gilbert, and J. Crisco. X-ray reconstructionof moving morphology (xromm): precision, accuracyand applications in comparative biomechanics research.Journal of Experimental Zoology Part A: EcologicalGenetics and Physiology, 313(5):262–279, 2010.

[2] Y. Ding, N. Gravish, C. Li, R. Maladen, N. Mazou-chova, S. Sharpe, P. Umbanhowar, and D. Goldman.

1 cm

(a) Sample frame

ATFATF TrajectoryFAFA Trajectory

(b) Tracking

0 5 10 15 2010

−2

100

102

Marker Id

Err

or: p

ixel

s

ATFFA

(c) Root mean square error per marker

800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

ATF

FA

Frames

(d) Correspondence validity vs time for Marker 10

Figure 5. Ocellated skink tracking.

Comparative studies reveal principles of movement onand within granular media.

[3] S. Gatesy, D. Baier, F. Jenkins, and K. Dial. Scientificrotoscoping: a morphology-based method of 3-d mo-tion analysis and visualization. Journal of ExperimentalZoology Part A: Ecological Genetics and Physiology,313(5):244–261, 2010.

[4] D. Haase, J. Nyakatura, and J. Denzler. Multi-view ac-tive appearance models for the x-ray based analysis ofavian bipedal locomotion. Pattern Recognition, pages11–20, 2011.

[5] T. Hedrick. Software techniques for two-and three-dimensional kinematic measurements of biological andbiomimetic systems. Bioinspiration & Biomimetics,3:034001, 2008.

[6] T. Hedrick, B. Tobalske, I. Ros, D. Warrick, andA. Biewener. Morphological and kinematic basis ofthe hummingbird flight stroke: scaling of flight muscletransmission ratio. Proceedings of the Royal Society B:Biological Sciences, 2011.

[7] R. Kalman et al. A new approach to linear filteringand prediction problems. Journal of basic Engineering,82(1):35–45, 1960.

[8] T. Lindeberg. Feature detection with automatic scaleselection. International journal of computer vision,30(2):79–116, 1998.

[9] R. Maladen, Y. Ding, C. Li, and D. Goldman. Undula-tory swimming in sand: Subsurface locomotion of thesandfish lizard. Science, 325(5938):314–318, 2009.

[10] R. Maladen, Y. Ding, P. Umbanhowar, A. Kamor, andD. Goldman. Mechanical models of sandfish locomo-tion reveal principles of high performance subsurfacesand-swimming. Journal of The Royal Society Inter-face, 8(62):1332–1345, 2011.

[11] J. Munkres. Algorithms for the assignment and trans-portation problems. Journal of the Society for Industrialand Applied Mathematics, 5(1):32–38, 1957.