TrackMPNN: A Message Passing Graph Neural Architecture forMulti-Object Tracking

Akshay Rangesh1, Pranav Maheshwari2, Mez Gebre2, Siddhesh Mhatre2, Vahid Ramezani2, andMohan M. Trivedi1

{arangesh, mtrived}@ucsd.edu, {pranav, mez, siddhesh, vahid.ramezani}@luminartech.com1Laboratory for Intelligent & Safe Automobiles, UC San Diego, CA

2Luminar Technologies, Inc., Palo Alto, CA

Abstract

This study follows many classical approaches to multi-object tracking (MOT) that model the problem using dy-namic graphical data structures, and adapts this formula-tion to make it amenable to modern neural networks. Ourmain contributions in this work are the creation of a frame-work based on dynamic undirected graphs that representthe data association problem over multiple timesteps, anda message passing graph neural network (MPNN) that op-erates on these graphs to produce the desired likelihood forevery association therein. We also provide solutions andpropositions for the computational problems that need tobe addressed to create a memory-efficient, real-time, onlinealgorithm that can reason over multiple timesteps, correctprevious mistakes, update beliefs, and handle missed/falsedetections. To demonstrate the efficacy of our approach,we only use the 2D box location and object category IDto construct the descriptor for each object instance. De-spite this, our model performs on par with state-of-the-artapproaches that make use of additional sensors, as well asmultiple hand-crafted and/or learned features. This illus-trates that given the right problem formulation and modeldesign, raw bounding boxes (and their kinematics) from anyoff-the-shelf detector are sufficient to achieve competitivetracking results on challenging MOT benchmarks.

1. Introduction

Object tracking is a crucial part of many complex au-tonomous systems and finds application in a variety ofdomains that require tracking of humans and/or objects.

‡Code: https://github.com/arangesh/TrackMPNN

Figure 1: Most tracking-by-detection approaches are re-duced to successive bipartite matching problems betweentwo disjoint sets, namely - sets of detections from two (con-secutive) timesteps as shown in the illustration above. Thegoal of the multi-object tracker is to propose correct associ-ations between the two sets (bold lines), and suppress spu-rious ones (gray lines).

Tracking introduces memory and persistence into a systemvia association, as opposed to standalone measurements anddetections in time. This enables systems to monitor objector agent behavior over a period of time, thus capturing his-torical context which can then be used to better assess cur-rent state or more accurately predict future states. Exam-ples of such usage from different domains include - gaug-ing human/pedestrian intent and attention based on historyof movement [6][19][25], surveillance of crowds scenes toperceive crowd behavior [30], predicting future trajectoriesof vehicles based on past tracks [29][5], tracking compactentities like cells and molecules to better understand biolog-ical processes [12] etc.

Tracking methods are numerous and varied in their ap-proach, and can broadly be categorized based on the number

arX

iv:2

101.

0420

6v4

[cs

.CV

] 7

May

202

1

of objects being tracked i.e. single or multi-object tracking(MOT) approaches. MOT approaches go beyond featureand appearance tracking and attempt to solve successivedata association problems to resolve conflicts between mul-tiple competing observations and tracks. Thus, the problemof MOT is also a problem of data association - where multi-ple observations are to be assigned to multiple active objecttracks to optimize some global criterion of choice. MOTapproaches can further be classified based on their mode ofoperation i.e. online, offline (batch), or near-online. Onlineapproaches provide tracks in real-time and continuously,and thus find use in real-time systems like autonomous cars.On the other hand, offline approaches benefit from viewingan entire sequence (i.e. conditioning on all available data)before estimating object tracks. This makes them relativelymore robust than their online counterparts. However, theseapproaches are only suited to applications where post hocoperation is not a hindrance, e.g. surveillance.

Modern MOT techniques typically work within thetracking-by-detection paradigm, where trackers function bystitching together individual detections across time (see Fig-ure 1). These detections can either be obtained from a sepa-rate upstream object detector that operates independently,or by integrating the detector and tracker into one cohe-sive unit. Irrespective of the particularities of the approach,multi-object trackers aim to robustly track multiple objectsthrough appearance changes, missed/false detections, tem-porary occlusions, birth of new tracks, death of existingtracks, and other such inhibitors. They do this by engineer-ing better feature descriptors for each object, improving thesimilarity or cost function used during data association, orby better optimizing the underlying objective.

In this study, we propose a novel MOT frameworkbased on dynamic, undirected, bipartite graphs for con-temporary neural networks. Our approach is partly moti-vated by the probabilistic graph representation of classicalmulti-target tracking methods like Multi-Hypothesis Track-ing (MHT) [4], with a focus on compatibility with mod-ern graph neural networks (GNNs). Our approach is on-line, capable of tracking multiple objects, reasoning overmultiple timesteps, and holding multiple hypotheses at anygiven time. In addition to this, our proposed approach canrun real-time on almost any modern GPU without exceed-ing memory and compute limits.

2. Related ResearchModern MOT techniques have different approaches to

address the underlying problem of data association. Whilemost methods utilize some form of tracking-by-detection,some create a cohesive framework where the detector andtracker work in tandem [37][13][38]. This usually leads toa symbiotic relationship, where detector and tracker ben-efit one another. In similar fashion, some studies have

explored the benefits of incorporating other related tasksin their MOT framework, including segmentation [35][32]and reconstruction[18]. Some others tweak classical ap-proaches by making them more tractable [11], or by devel-oping better feature representations [15] - thereby makingthem amenable for deployment in dense, cluttered scenes.Other works focus solely on feature engineering and tun-ing of cost functions [27, 14], and assume detections froma trained off-the-shelf detector. Finally, with the introduc-tion of large scale datasets for autonomous driving, manymodern approaches tend to make use of 3D sensors like Li-DARs, either by itself [21], or via multi-modal fusion withother imaging sensors like cameras [22]. This helps thesemethods resolve temporary occlusion, truncation, and othersuch issues arising from 3D ambiguity.

In this study, we introduce a MOT framework suitablefor Graph Neural Networks (GNNs). GNNs have found awide range of applications in recent years; see [7] and ref-erences therein. GNNs can be viewed as a generalizationof convolutional neural networks(CNNs) [20]. Multi-layerCNNs can extract multi-scale features from spatio-temporaldata and generate representations which can solve a vari-ety of machine learning tasks. However, CNNs are definedby kernels that operate only on regular multi-dimensionalgrids. GNNs generalize CNNs kernel operation to generalgraphs (of which CNNs on regular grids are special cases).More importantly for us, one can interpret the kernel oper-ations in this graphical context as generating an output ateach node using the values of neighboring nodes and thevalue of the node itself as inputs. This gives rise to a Mes-sage Passing Neural Network (MPNN) architecture [9]. Aswe shall see in the following sections, our architecture isessentially an MPNN with communicating memory unitswhose graphical structure evolves in time. This kind of ar-chitecture, a GNN with communicating memory units, hasbeen applied to the modeling of spatially interacting mobileagents in [26]. GNNs in general have recently found usein multi-object tracking [33, 34, 16]; these models however,do not have the ability to pass messages to other parts ingraph, and simply use the graph structure to infer similarityscores between consecutive sets of detections. We know ofonly one other approach which applies GNNs with messagepassing to the multi-object tracking problem [2]. Unlike [2]where the model is run on batches of detections in an of-fline manner (with score averaging), the proposed approachinvolves dynamic graphs which are processed in a rolling-window scheme - making it suitable for online, real-timeapplications. Moreover, our message passing operations arespecifically crafted with rapidly evolving graph structures inmind.

The proposed approach is also partly motivated by theprobabilistic graph representation of classical multi-targettracking methods, for example hypothesis based and track-

Figure 2: Our proposed MOT framework is based on dynamic undirected graphs that depict the data association problemover multiple timesteps. In the Figure above, dji represents a detection node indexed by i, at timestep j.

tree based MHT (Multi-Hypothesis Tracking) [4][23]. Inthe these methods, the association of observations to tar-gets/tracks is resolved by generating branches emanatingfrom existing global hypotheses to account for new observa-tions, then propagating the posterior probability of resultingglobal hypotheses, recursively, along those branches. As inthe earlier hypothesis based classical approaches, we do notgenerate explicit track-trees representing a set of underly-ing targets and we limit the size of the active section of thegraph determining the associations to prevent exponentialgrowth of compute. We also make no explicit assumptionsabout the underlying Bayesian distributions. The networklearns from the ground truth tracking data, how best to gen-erate the tracks during inference. Also, we do not enumeratethe set of consistent tracks. The graph represents the super-position of all such tracks which can be decoded to producethe optimal one.

3. Proposed Tracking FrameworkThis approach works based on the following idea: as

data association in multi-object tracking (MOT) is conven-tionally modelled as a bipartite graph over two consecutive

timesteps, could we simply learn and infer directly on suchgraph structures using GNNs? This graph structure couldalso be expanded to cover multiple consecutive timestepsso as to incorporate non-local information and infer distantassociations.

Our approach formulates the MOT problem as infer-ence on an undirected graph where each detection is rep-resented as a node in the graph, and potential associationsbetween different detections are represented by edges con-necting them. The graph is dynamically updated at everynew timestep, where new detections (and associations) fromthe latest timestep are added, and old, inactive detections(and associations) are removed. An overview of this setupis illustrated in Figure 2. The proposed model (described inSection 4) works by operating on this dynamic graph in arolling window basis (from left to right as presented in Fig-ure 2). The size of the rolling window is a design choice tobe made based on the desired performance, available mem-ory and compute requirements. This concept of a rollingwindow is similar to the one used in [3].

Although not explicitly drawn in Figure 2 in order to re-duce clutter, each edge joining two detection nodes is also

treated as a node of the graph in actuality. This helps us en-dow a vector representation to each potential association be-tween two detections. Thus, this dynamic graph structure isbipartite, with detection nodes and association nodes form-ing the two disjoint and independent sets. Detection nodesrepresent object detections in a sequence, and are initializedwith their corresponding feature descriptors. Associationnodes represent potential pairwise associations between twodetections from different timesteps, and are initialized withzero vectors. These initializations are then transformed andupdated by the model to create hidden representations ofevery node in the graph with each additional timestep.

3.1. Training & Inference

Before we describe the model architecture and optimiza-tion in detail (Section 4), we provide a macro view of thetraining and inference procedure that we adopt. We makeuse of the following high level functions pertaining to thegraph structure and the model during both training and in-ference:

• initialize graph(): This creates an initial bipartitegraph with detection nodes from two consecutivetimesteps, with an association node between every de-tection pair.

• update graph(): This function is called at the start ofevery new timestep, to add new (detection and associ-ation) nodes and edges to end of the currently activegraph. Note that association nodes are only added be-tween the newly introduced detection nodes and un-paired detection nodes from previous timesteps. Thisfunction also removes the oldest set of nodes and edgesfrom the currently active part of the graph. It essen-tially moves the rolling window one step forward.

• prune graph(): This function removes low probabilityedges and nodes from the currently active part of thegraph using a user specified confidence threshold. Thisfunction can be called whenever memory/compute re-quirements exceed what is permissible, or to preventan explosion of nodes.

• decode graph(): This function is called to decode themodel-produced output probabilities for every node inthe graph into corresponding object tracks. This caneither be done in a greedy manner (by following thehighest probability path from left to right) or by us-ing the Hungarian algorithm (on consecutive timestepsfrom left to right).

• TrackMPNN(): This initializes an instance of the pro-posed model (described in Section 4).

• TrackMPNN.forward(): This carries out one forwardpass of the data through the model.

• TrackMPNN.backward(): This carries out one back-ward pass through the model to produce gradients withrespect to the losses.

The training and inference pseudo-code that make use ofthese functions are presented in Algorithms 1 and 2. As de-scribed earlier, the inference procedure operates in a rollingwindow manner, from past (left) to future (right). Thus,we operate on the entire tracking sequence in a continu-ous manner during inference. However, this is not possi-ble during training, where gradients need to be stored forall detection and association nodes encountered in the dy-namic graph. To make the training process computation-ally tractable, we split each tracking sequence into smallercontiguous chunks, and process each such mini-sequence asan individual sample. As depicted in Algorithm 1, we alsoaccumulate losses over the sequence length and backpropa-gate only after the entire sequence has been processed.

4. Model ArchitectureIn this study, we use a class of Graph Neural Networks

called Message Passing Neural Networks (MPNNs) [9].Like most MPNNs, our proposed TrackMPNN model con-sists of:

• a message function:

#»mkv =

∑w∈N (v)

M(#»

h k−1v ,

#»

h k−1w ), (1)

• a node/vertex update function:

#»

h kv = U(

#»

h k−1v , #»mk

v), (2)

• a readout/output function:

ykv = R(#»

h kv), (3)

where vertices v, w are nodes in graph G, and in our con-text be either detection nodes or association nodes. Notethat iteration k is different from the current timestep t ofthe tracking sequence. Whereas t signifies the lifetime ofthe entire graph, k denotes the lifetime of each individualnode in this dynamic graph. For our purposes, we use sep-arate functions/weights for detection nodes and associationnodes, which we detail in the next two subsections.

4.1. Detection node operations

Consider the illustration of a detection node d and itsneighboring association nodes ai ∈ N (d) presented in Fig-ure 3. The operations associated with nodes of this type arepresented below.

Algorithm 1 Pseudo-code for one training iterationInput: feats, labelsnet← TrackMPNN() . initialize TrackMPNN modelG← initialize graph(feats) . initialize graph with detection features and pairwise associations from first two timestepstotal loss← 0 . initialize loss to 0(probabilities, loss)← net.forward(G, labels) . forward passtotal loss← total loss+ loss . add to total lossfor t← 2 to tend do

G← update graph(G, feats, t) . add new nodes and edges to graph; remove earliest nodes and edges(probabilities, loss)← net.forward(G, labels) . forward passtotal loss← total loss+ loss . add to total lossif condition then

G← prune graph(G, probabilities) . remove low probability nodes to limit memory footprintend if

end fornet← net.backward(total loss) . backward pass to train modelreturn net

Algorithm 2 Pseudo-code for inference on a MOT sequenceInput: featstracks← {} . initialize tracks to emptynet← TrackMPNN(trained weights) . initialize TrackMPNN model with trained weightsG← initialize graph(feats) . initialize graph with detection features and pairwise associations from first two timestepsprobabilities← net(G) . forward pass to get probabilitiestracks← tracks

⋃decode graph(G, probabilities) . decode model probabilities to produce tracks for desired window

for t← 2 to tend doG← update graph(G, feats, t) . add new nodes and edges to graph; remove earliest nodes and edgesprobabilities← net(G) . forward pass to get probabilitiestracks← tracks

⋃decode graph(G, probabilities) . decode model probabilities to produce tracks for desired window

if condition thenG← prune graph(G, probabilities) . remove low probability nodes to limit memory footprint

end ifend forreturn tracks

4.1.1 Initialization

The hidden state of a detection node is initialized with a pairof linear transformations (with non-linearity and normaliza-tion) of its input feature representation #»x d:

#»

h 0d = Wi′

det BatchNorm(#»

x′d) +#»

b i′

det, (4)

where#»

x′d = ReLU(Widet

#»x d +#»

b idet). (5)

The input representation #»x d of a detection node d canbe composed of different features and attributes describingthe detected object instance (a detection produced by theobject detector). In this study, we use detections producedby a recurrent rolling convolutions (RRC) detector [24] -though any other detector could be used instead. Contraryto many contemporary approaches, we only use the 2D boxlocations and object categories produced by the detector todescribe each object instance, and rely on the model to learnthe patterns of sequential locations on the image plane.

The input representation of each detection #»x d is then de-fined as follows:

#»x d = [ #»x d;2d|| #»x d;cat], (6)

where || represents concatenation along the singleton di-mension.

#»x d;2d = (x1, y1, x2 − x1, y2 − y1, score)ᵀ (7)

denotes 2D features comprised of the 2D bounding box lo-cation (top-left location, width, height) and score, and,

#»x d;cat = (0, · · · , 1, · · · , 0)ᵀ (8)

is the one-hot encoding of the object category.

4.1.2 Message and node/vertex update functions

The hidden state of the detection node is updated based onits previous hidden state and the previous hidden states of

Figure 3: Illustration of a detection node d and its neighbor-ing association nodes {ai|ai ∈ N (d)}. The neighborhoodcan be split into two disjoint sets N−(d) and N+(d), de-noting all nodes from the past and future respectively.

its neighboring association nodes, weighted by an attentionmechanism that takes into account the detection nodes con-nected by the association:

#»

h kd = GRU

( #»

h k−1d , #»mk

d

)= GRU

(#»

h k−1d ,

∑ai∈N (d)

αk−1dai

#»

h k−1ai

),

(9)

where #»mkd is the message passed to detection node d at iter-

ation k, and αk−1dai

are the attention weights [31] assigned toeach association node ak−1i .

Fully expanded out, the coefficients computed by the at-tention mechanism may then be expressed as:

αk−1dai

=exp(LReLU(aᵀ|Wh

det

#»h k−1

d −Whdet

#»h k−1

di|))∑

dj∈N2(d) exp(LReLU(aᵀ|Whdet

#»h k−1

d −Whdet

#»h k−1

dj|)),

(10)where di ∈ N (ai) and di 6= d, N 2(·) represents thesecond-order neighborhood of a node, and LReLU denotesthe LeakyReLU non-linearity (with negative input slope0.2).

4.1.3 Readout function

The detection node output okd represents a scalar confidencevalue obtained by simple linear transformation of its hiddenstate:

okd = Wodet

#»

h kd +

#»

b odet. (11)

4.2. Association node operations

Consider the illustration of an association node a and itsneighboring detection nodes depicted in Figure 4. The op-erations associated with nodes of this type are presented be-low.

Figure 4: Illustration of an association node a and its twoneighboring detection nodes {d1, d2}.

4.2.1 Initialization

The hidden state of an association node is initialized with avector of of 0s.

#»

h 0a =

#»0 (12)

4.2.2 Message and node/vertex update functions

The hidden state of the association node is updated based onits previous hidden state and the previous hidden states of itstwo neighboring detection nodes. Given the fixed degree ofan association node, we experiment with two different mes-sage update functions. The first is based on the differencebetween the hidden states of its neighbors:

#»

h ka = GRU

( #»

h k−1a , #»mk

a

)= GRU

(#»

h k−1a ,Wh

ass[#»

h k−1d2− #»

h k−1d1

] +#»

b hass

),

(13)

and the second is based on their concatenation:

#»

h ka = GRU

( #»

h k−1a , #»mk

a

)= GRU

(#»

h k−1a ,Wh

ass[#»

h k−1d1|| #»h k−1

d2] +

#»

b hass

).

(14)

In both cases, #»mka is the message passed to association node

a at iteration k.

4.2.3 Readout function

The association node output oka represents a scalar confi-dence value obtained by simple linear transformation of itshidden state:

oka = Woass

#»

h ka +

#»

b oass. (15)

4.3. Training Losses

Let G = (V,E) denote the dynamic graph at any giveninstant. We can further split the set of vertices into twodisjoint sets, i.e. V = DN ∪ AN and DN ∩ AN = ∅,where DN and AN represent the set of detection nodesand association nodes respectively.

We first apply a binary cross-entropy loss at each detec-tion node:

Lbce;DN = − 1

|DN |∑

d∈DN

(yd log(Sigmoid(okd

d ))

+ (1− yd) log(1− Sigmoid(okd

d ))

), (16)

where kd denotes the latest iteration of detection node d,and

yd =

{1, if true positive0, otherwise.

(17)

Similarly, we apply a binary cross-entropy loss at each as-sociation node:

Lbce;AN = − 1

|AN |∑

a∈AN

(ya log(Sigmoid(oka

a ))

+ (1− ya) log(1− Sigmoid(okaa ))

), (18)

where ka denotes the latest iteration of association node a,and

ya =

{1, if N (a) belongs to the same track0, otherwise.

(19)

We also apply a cross-entropy loss for every set of compet-ing association nodes:

Lce;AN = − 1

|DN |∑

d∈DN

(1

|N−(d)|∑

a∈N−(d)

ya log(Softmax(okaa ))

+1

|N+(d)|∑

a∈N+(d)

ya log(Softmax(okaa ))

), (20)

where

ya =

{1, if N (a) belongs to the same track0, otherwise,

(21)

N−(·) is the set of all neighbors from past timesteps, andN+(·) is the set of all neighbors from future timesteps.

The total loss used to train the entire model is the sum ofthe individual losses:

L = Lbce;DN + Lbce;AN + Lce;AN (22)

5. Experiments and Analyses5.1. Implementation Details

Dynamic Undirected Graph: As detailed in Section 3,our approach operates using a temporal rolling window,

where detection and association nodes falling inside thewindow make up the dynamic graph at any given instant.The two hyperparameters that govern this process are thecurrent window size (CWS) and the retained window size(RWS). CWS defines the size of the rolling window in dis-crete timesteps/frames. RWS determines the number of dis-crete timesteps for which an unassociated detection node iskept in the dynamic graph after it is no longer in the rollingwindow. This is to ensure that objects that are temporar-ily occluded or missing have a chance to be re-identifiedagain. The rolling window is updated after each timestepby adding new nodes from the next timestep and removingnodes from the earliest timestep. Before detection nodesare removed however, they are assigned to tracks as part ofthe decoding process i.e., for every detection node that isremoved, it is assigned to the same track as an earlier detec-tion node corresponding to the maximum association prob-ability as produced by the model (Eq. 15). If a true positivedetection node does not have any high probability associa-tions, a new track is initialized.Training & Optimization: As depicted in Eq. 22, the to-tal loss is simply the sum of the individual losses - with-out any weighting. We also forego mini-batch training andopt for mini-sequence training in its place. Instead of us-ing a mini-batch of examples for each training sample, weuse one mini-sequence for each training sample. A mini-sequence is simply a tracking sequence of CWS contiguoustimesteps, randomly sampled from full-length tracking se-quences. Similar to mini-batch gradient descent, the lossesare accumulated, and then backpropagated after the entiremini-sequence is processed. In our experience, this tends tostabilize training, produce better gradients, and require lessmemory. The entire network is trained using an Adam opti-mizer with learning rate 0.0001, β1 = 0.9 and β2 = 0.999for 50 epochs. Additionally, we augment the training splitwith multiple random transformations. This includes timereversal (reverse timestep/frame ordering), dropout of de-tections (ignore a fraction of true positive detections), andhorizontal flips. Finally, we initialize the bias values in thedetection and association node readout functions (Eq. 11,15) to +4.595 and −4.595 respectively. This is to keep thelosses manageable during the initial phase of training.Inference: During inference, care is taken to ensure thathyperparameters associated with the dynamic graph remainunchanged from training. Instead of feeding the modelmini-sequences, we instead supply the full-length sequenceas a single sample. The model operates on this sequencein a rolling window manner, generating continuous tracksalong the way.

5.2. Ablation Experiments

To perform ablation experiments, we make use of theKITTI MOT dataset [8]. The KITTI MOT dataset is com-

(a) MOTA versusCurrent WindowSize (RWS=0)

(b) MOTA versusRetained Window

Size (CWS=5)

(c) MOTA versusHidden State Size(CWS=5, RWS=0)

Figure 5: Experiments illustrating the effect of differentmodel settings on the tracking performance using the KITTIvalidation split (for Cars only).

prised of 21 training sequences and 29 testing sequences,with two categories of interest: Cars and Pedestrians. Forthe purpose of ablation, we use the first 11 training se-quences to train the models, and leave the rest for valida-tion. We also restrict our experiments to one object category- Cars. For evaluation, we make use of the CLEAR MOTmetrics [1, 17]. In cases where only one metric is desired,we use the Multi-Object Tracking Accuracy (MOTA). Thefollowing paragraphs detail our ablation of different settingspertaining to the dynamic graph and the model.Dynamic graph settings: To assess the benefits of a poten-tially larger rolling window size, we trained multiple mod-els with different settings of CWS. In all cases, RWS was setto 0. Figure 5a depicts the plot of MOTA for different set-tings of CWS. Although all variants achieve very similar re-sults, CWS=10 produces the best MOTA. CWS=5 achievesvery similar results at a lower computational and memorycost. Next, we fix CWS=5 and evaluate models with dif-ferent values of RWS. The results from this experiment areplotted in Figure 5b. Once again, we notice that the MOTAvalues are quite similar, with RWS=5 achieving the best re-sults. Both these experiments also highlight the robustnessof the model across various graph settings.Model settings: To compare different model settings andtheir impacts on performance, we first define a baselinemodel (B) against which other variants can be compared.The baseline model is trained using graph settings CWS=5,RWS=0, and uses difference based updates for the asso-ciation nodes (Eq. 13). Results from these comparisonsare presented in Table 1. First, we measure the efficacyof the greedy matching procedure by comparing it with amodel using the Hungarian algorithm for optimal bipar-tite matching (B w/ H). In this variant, the Hungarian al-gorithm is used to decode the graph into tracks, by opti-mally matching detections to existing tracks based on as-sociation probabilities. We notice that the results are bet-ter across the board. Thus, the Hungarian algorithm helpsproduce slightly longer and continuous tracks - at the ex-pense of additional computational cost. Second, we traina model (B w/o TP) without true-positive classification at

Table 1: Results from experiments on the KITTI validation set (forCars only). Arrows indicate if a higher (↑) or lower value of themetric (↓) is desired.

Model type MOTA(↑)

MOTP(↑)

MT(↑)

ML(↓)

IDS(↓)

FRAG(↓)

B1 84.69 0.1009 71.49% 2.26% 320 156B w/ H2 85.99 0.1012 73.20% 2.26% 140 152

B w/o TP3 85.18 0.1013 72.85% 2.26% 239 153B||4 79.30 0.1108 65.71% 5.61% 703 205

B w/ RT5 84.86 0.1008 72.85% 2.26% 291 1531 B: baseline model with difference based updates for associationnodes 2 H: Hungarian algorithm 3 TP: true positive classification4 B||: baseline model with concatenation based updates for associa-tion nodes (Eq. 14) 5 RT: random transformations during training

the detection nodes (see Eq. 16). In this case, every de-tection is assumed to be a true positive, and the model isused to only produce association probabilities. From theresults presented in Table 5, we see that this setting alsoimproves performance on all metrics. This makes sensebecause 2D box locations alone are usually not enough todistinguish true positives from false positives. Third, tocompare the two different update functions for associationnodes, we also train a variant of the baseline with concate-nation based updates (Eq. 14). When compared to the dif-ference based updates in B, concatenation produces worseresults with a much lower MOTA and a higher IDS. Thisindicates that difference based updates can better model thesimilarity/dissimilarity between two detection nodes. Next,to quantify the effects of data augmentation during training,we train a model with random transformations (B w/ RT).This too improves most metrics across the board - indicat-ing the clear benefits of our augmentation scheme - espe-cially when trained on datasets of limited size. Finally, togauge the effects of hidden state size on the tracking per-formance, we train multiple models containing GRUs withdifferent hidden state sizes. The resulting MOTAs for dif-ferent variants are plotted in Figure 5c. The plot indicatesthat increasing the dimensionality of the hidden state tendsto improve performance, but this trend saturates beyond 64dimensions.

5.3. Comparison with State of the Art

Model and graph settings: To compare our proposedapproach with existing methods, we train a TrackMPNNmodel with CWS=5, and incorporate random transforma-tions for data augmentation. We also discard true positiveclassification, and use the Hungarian algorithm during in-ference. This model is trained with the same optimizer set-tings as the baseline for a total of 30 epochs, using 18 of the21 training sequences provided in KITTI MOT. The remain-ing 3 sequences were used for validation and early stopping.KITTI MOT benchmark: We compare two variants ofour approach with other competing methods on the KITTI

Table 2: Results on the KITTI multi-object tracking benchmark for Cars. Arrows indicate if a higher (↑) or lower value ofthe metric (↓) is desired. Our results are in bold.

Method Sensors Online MOTA(↑)

MOTP(↑)

MT(↑)

ML(↓)

IDS(↓)

FRAG(↓)

CenterTrack [38] RGB camera 31 0.8883 0.8497 82.15 % 2.46 % 254 227TrackMPNN + CenterTrack RGB camera 3 0.8733 0.8449 84.46 % 2.15 % 481 237

JRMOT [28] RGB camera, LiDAR 3 0.8510 0.8528 70.92 % 4.62 % 271 273TrackMPNN + RRC RGB camera 3 0.8455 0.8507 70.92 % 4.00 % 466 482

mono3DT [13] RGB camera, GPS 3 0.8428 0.8545 73.08 % 2.92 % 379 573MOTSFusion [18] RGB camera, stereo 3 0.8424 0.8503 72.77 % 2.92 % 415 569

SMAT [10] RGB camera 3 0.8364 0.8589 62.77 % 6.00 % 198 294mmMOT [36] RGB camera 52 0.8323 0.8503 72.92 % 2.92 % 733 570

MOTBeyondPixels [27] RGB camera 3 0.8268 0.8550 72.61 % 2.92 % 934 5811 online/near-online 2 offline/batch

Table 3: Results on the KITTI multi-object tracking benchmark for Pedestrians. Arrows indicate if a higher (↑) orlower value of the metric (↓) is desired. Our results are in bold.

Method Sensors Online MOTA(↑)

MOTP(↑)

MT(↑)

ML(↓)

IDS(↓)

FRAG(↓)

CenterTrack [38] RGB camera 3 0.5384 0.7372 35.40 % 21.31 % 425 618TrackMPNN + CenterTrack RGB camera 3 0.5210 0.7342 35.05 % 18.90 % 626 6691 online/near-online 2 offline/batch

MOT benchmark in Table 2. One variant makes use ofRRC detections [24] (TrackMPNN + RRC), and the otheruses detections produced by the CenterTrack tracker [38](TrackMPNN + CenterTrack). Our TrackMPNN + RRCmodel is trained to track Cars, whereas TrackMPNN + Cen-terTrack is trained to track all object categories on KITTIi.e., Cars, Pedestrians and Cyclists. First, our TrackMPNN+ RRC model outperforms all competing 2D methods thatmake use of the same set of detections [14, 13, 18, 10,36, 27] on the MOTA metric, and remains competitive onother metrics. For RRC detections, we only fall short of JR-MOT [28] which makes use of LiDAR point clouds for 3Drange information. Next, when using the same set of detec-tions as CenterTrack, we beat every other approach and arecomparable in terms of performance to the state-of-the-art.Once again, we do this without relying on any visual cues orpre-training on other datasets as in [38]. The performanceof our TrackMPNN + CenterTrack model is also compara-ble to [38] for tracking Pedestrians (see Table 3) when usingthe same set of detections. While we fall short of Center-Track on a few metrics, we beat them in some others (MTand ML).Qualitative results from KITTI MOT: In addition to thequantitative results presented above, we also show somequalitative results of our method from the KITTI MOT test-ing set in Figure 6. The results depict multiple examplesof tracking across frames in a sequence, overlaid with de-tection boxes. The boxes are color coded to indicate eachunique track produced by our model.

5.4. Analyzing Learned Attention Weights

As described in Section 4.1.2 of the paper, we makeuse of graph attention for detection nodes d to selec-tively receive messages from neighboring association nodes{ai|ai ∈ N (d)}. However, we do not enforce any restric-tions on which associations in the neighborhood should re-ceive more/less weights. This is learned by the model whenoptimizing for the losses during training.

To better understand the nature of these attention weightsassigned by the model, we train a baseline model B withmulti-head attention comprised of three attention heads. Weuse the same settings and train-val split as the ablation ex-periments. The trained model is then run on the validationset, and the attention weights assigned to every associationnode are grouped based on whether the underlying associa-tion is true or false, i.e. if the neighboring detections con-nected by the association node belong to the same track ornot.

The distributions of these two separate sets of attentionweights (for each of three attention heads) are presented inFigure 7. First, we see that higher attention weights aregenerally assigned to true associations, while those for in-correct associations are skewed towards 0. This implies thatmessages are mostly exchanged between detection nodesbelonging to the same track. Next, we notice different dis-tributions for each attention head, indicating that each headtends to receive different messages from the neighborhood.This is especially true for the second attention head in Fig-ure 7. This experiment clearly indicates that the modellearns to weight the messages appropriately without any ex-plicit supervision.

Figure 6: Qualitative examples on the KITTI multi-object tracking (MOT) testing set. Each row depicts a sequence of frames,with overlaid color-coded detection boxes. Each color represents a unique track generated by our model.

Figure 7: Histogram plots of the attention weights assignedto true (left column) and false associations (right column)for each of three attention heads. Each row above corre-sponds to one of three attention heads.

5.5. Compute Requirements and Runtime

Since this approach relies on dynamic graphs and opera-tions on them, it is does not have a fixed memory allocation

or runtime per timestep. These numbers can change de-pending on the size and connectivity of the dynamic graph- which in turn depends on the number of objects in thescene and the associations made in the recent past. To keepthe memory requirements to a minimum, we make use ofsparse operations wherever possible - especially in our ver-tex update functions. To understand the memory require-ments of the model described in Section 5.3, we plot his-tograms of allocated GPU memory during training and in-ference (Figure 8a, 8b). From these plots, we notice thatdespite some spread, the allocated memory remains wellwithin reasonable limits during training and inference (550-900MB). This makes it possible to train and test our modelon most modern desktop and laptop GPUs.

Similarly, we plot a histogram of our model runtime onthe KITTI MOT test set (Figure 8c) using an NVIDIA Ti-tan X Maxwell GPU. This plot demonstrates that our entireframework is capable of operating in real-time, taking only0.01 seconds on average to process an entire timestep. Fi-nally, to examine the effects of the size of the dynamic graphon runtime, we plot the number of detections nodes in thegraph versus the observed runtime in Figure 8d. We canclearly see a monotonically increasing relationship betweenthe two; but the runtime increases at a faster rate as moredetection nodes are added. This does not pose a problemfor the KITTI dataset, but for larger datasets with highlycluttered scenes, occasional pruning operations can be usedto keep the memory allocation within bounds.

(a) Memory allocated duringtraining

(b) Memory allocated duringinference

(c) Inference runtime (d) Number of detection nodesversus runtime

Figure 8: Plots examining the compute requirements and runtime of our proposed approach when using RRC detections onthe KITTI MOT dataset.

6. Concluding Remarks

This study proposes a tracking framework based onundirected graphs that represent the data association prob-lem over multiple timesteps in the tracking-by-detectionparadigm. In this framework, both individual detections andassociations between pairs of them are represented as nodesin the graph. Furthermore, the graph is dynamic, whereonly detections (and their associations) within a certain tem-poral range are processed in a rolling window basis. Thisform of data representation offers any multi-object trackingmodel the following benefits - multiple competing associa-tions can be considered while scoring any given associationfrom two detections, including associations over multipletimesteps. Information can be stored and propagated acrossmany timesteps through message passing operations. Mis-takes in the past can be corrected as long as they are stillpart of the dynamic graph. False positives, duplicate andmissed detections can be handled intrinsically as part of themodel.

To illustrate these benefits, we also present a messagepassing neural network (TrackMPNN) that operates onthese dynamic undirected graphs, and train it on the KITTIdataset. Our proposed training and inference schemes makethis possible with limited memory and computational band-width, and enable real-time operation despite large numberof objects in the scene. Experiments, qualitative examplesand competitive results on popular MOT benchmarks forautonomous driving demonstrate the promise and unique-ness of the proposed approach.

7. Acknowledgements

We are grateful to the Laboratory for Intelligent & SafeAutomobiles at UC San Diego for providing us with theresources and compute to run the experiments presented inthe paper.

References[1] Keni Bernardin and Rainer Stiefelhagen. Evaluating mul-

tiple object tracking performance: the clear mot metrics.EURASIP Journal on Image and Video Processing, 2008:1–10, 2008. 8

[2] Guillem Braso and Laura Leal-Taixe. Learning a neu-ral solver for multiple object tracking. In Proceedings ofthe IEEE/CVF Conference on Computer Vision and PatternRecognition, pages 6247–6257, 2020. 2

[3] Wongun Choi. Near-online multi-target tracking with aggre-gated local flow descriptor. In Proceedings of the IEEE Inter-national Conference on Computer Vision, pages 3029–3037,2015. 3

[4] I. J. Cox and S. L. Hingorani. An efficient implementation ofreid’s multiple hypothesis tracking algorithm and its evalua-tion for the purpose of visual tracking. In IEEE Transactionson Pattern Analysis and Machine Intelligence, pages 138 –150, 1996. 2, 3

[5] Nachiket Deo, Akshay Rangesh, and Mohan M Trivedi. Howwould surround vehicles move? a unified framework for ma-neuver classification and motion prediction. IEEE Transac-tions on Intelligent Vehicles, 3(2):129–140, 2018. 1

[6] Nachiket Deo and Mohan M Trivedi. Trajectory forecastsin unknown environments conditioned on grid-based plans.arXiv preprint arXiv:2001.00735, 2020. 1

[7] Jie Zhou et al. Graph neural networks: A review of methodsand applications. In https://arxiv.org/pdf/1812.08434.pdf,2019. 2

[8] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are weready for autonomous driving? the kitti vision benchmarksuite. In Conference on Computer Vision and Pattern Recog-nition (CVPR), 2012. 7

[9] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E.Dahl. Neural message passing for quantum chemistry. InarXiv:1704.01212, 2017. 2, 4

[10] Nicolas Franco Gonzalez, Andres Ospina, and PhilippeCalvez. Smat: Smart multiple affinity metrics for multipleobject tracking. In Aurelio Campilho, Fakhri Karray, andZhou Wang, editors, Image Analysis and Recognition, pages48–62, Cham, 2020. Springer International Publishing. 9

[11] Seyed Hamid Rezatofighi, Anton Milan, Zhen Zhang, Qin-feng Shi, Anthony Dick, and Ian Reid. Joint probabilistic

data association revisited. In Proceedings of the IEEE inter-national conference on computer vision, pages 3047–3055,2015. 2

[12] Tao He, Hua Mao, Jixiang Guo, and Zhang Yi. Cell trackingusing deep neural networks with multi-task learning. Imageand Vision Computing, 60:142–153, 2017. 1

[13] Hou-Ning Hu, Qi-Zhi Cai, Dequan Wang, Ji Lin, Min Sun,Philipp Krahenbuhl, Trevor Darrell, and Fisher Yu. Jointmonocular 3d vehicle detection and tracking. In Interna-tional Conference on Computer Vision (ICCV), 2019. 2, 9

[14] Hasith Karunasekera, Han Wang, and Handuo Zhang. Mul-tiple object tracking with attention to appearance, structure,motion and size. IEEE Access, 2019. 2, 9

[15] Chanho Kim, Fuxin Li, Arridhana Ciptadi, and James MRehg. Multiple hypothesis tracking revisited. In Proceed-ings of the IEEE international conference on computer vi-sion, pages 4696–4704, 2015. 2

[16] Jiahe Li, Xu Gao, and Tingting Jiang. Graph networks formultiple object tracking. In Proceedings of the IEEE/CVFWinter Conference on Applications of Computer Vision,pages 719–728, 2020. 2

[17] Yuan Li, Chang Huang, and Ram Nevatia. Learning to asso-ciate: Hybridboosted multi-target tracker for crowded scene.In 2009 IEEE Conference on Computer Vision and PatternRecognition, pages 2953–2960. IEEE, 2009. 8

[18] Jonathon Luiten, Tobias Fischer, and Bastian Leibe. Trackto reconstruct and reconstruct to track. arXiv preprintarXiv:1910.00130, 2019. 2, 9

[19] Eshed Ohn-Bar and Mohan Manubhai Trivedi. Looking athumans in the age of self-driving and highly automated ve-hicles. IEEE Transactions on Intelligent Vehicles, 1(1):90–104, 2016. 1

[20] K. O’Shea and R. Nash. An introduction to convolutionalneural networks. In arXiv preprint arXiv:1511.08458, 2015.2

[21] Johannes Poschmann, Tim Pfeifer, and Peter Protzel. Factorgraph based 3d multi-object tracking in point clouds. arXivpreprint arXiv:2008.05309, 2020. 2

[22] Akshay Rangesh and Mohan Manubhai Trivedi. No blindspots: Full-surround multi-object tracking for autonomousvehicles using cameras and lidars. IEEE Transactions onIntelligent Vehicles, 4(4):588–599, 2019. 2

[23] D.B. Reid. An algorithm for tracking multiple targets. InIEEE Transactions on Pattern Analysis and Machine Intelli-gence, pages 843–854, 1979. 3

[24] Jimmy Ren, Xiaohao Chen, Jianbo Liu, Wenxiu Sun, Jia-hao Pang, Qiong Yan, Yu-Wing Tai, and Li Xu. Accuratesingle stage detector using recurrent rolling convolution. InProceedings of the IEEE conference on computer vision andpattern recognition, pages 5420–5428, 2017. 5, 9

[25] Daniela Ridel, Eike Rehder, Martin Lauer, Christoph Stiller,and Denis Wolf. A literature review on the prediction ofpedestrian behavior in urban scenarios. In 2018 21st Inter-national Conference on Intelligent Transportation Systems(ITSC), pages 3105–3112. IEEE, 2018. 1

[26] Renjie Liao Sergio Casas, Cole Gulino and Raquel Urta-sun. SpAGNN spatially-aware graph neural networks for

relational behavior forecasting from sensor data. In 2020IEEE International Conference on Robotics and Automation(ICRA), 2020. 2

[27] Sarthak Sharma, Junaid Ahmed Ansari, J. Krishna Murthy,and K. Madhava Krishna. Beyond pixels: Leveraging geom-etry and shape cues for online multi-object tracking. In Pro-ceedings of the IEEE International Conference on Roboticsand Automation (ICRA), 2018. 2, 9

[28] Abhijeet Shenoi, Mihir Patel, JunYoung Gwak, PatrickGoebel, Amir Sadeghian, Hamid Rezatofighi, RobertoMartın-Martın, and Silvio Savarese. Jrmot: A real-time 3dmulti-object tracker and a new large-scale dataset. In TheIEEE/RSJ International Conference on Intelligent Robotsand Systems (IROS), 2020. 9

[29] Sayanan Sivaraman and Mohan Manubhai Trivedi. Look-ing at vehicles on the road: A survey of vision-based vehi-cle detection, tracking, and behavior analysis. IEEE trans-actions on intelligent transportation systems, 14(4):1773–1795, 2013. 1

[30] HY Swathi, G Shivakumar, and HS Mohana. Crowd behav-ior analysis: A survey. In 2017 international conference onrecent advances in electronics and communication technol-ogy (ICRAECT), pages 169–178. IEEE, 2017. 1

[31] Petar Velickovic, Guillem Cucurull, Arantxa Casanova,Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph at-tention networks. arXiv preprint arXiv:1710.10903, 2017.6

[32] Paul Voigtlaender, Michael Krause, Aljosa Osep, JonathonLuiten, Berin Balachandar Gnana Sekar, Andreas Geiger,and Bastian Leibe. Mots: Multi-object tracking and seg-mentation. In Conference on Computer Vision and PatternRecognition (CVPR), 2019. 2

[33] X. Weng, Y. Yuan, and K. Kitani. Joint 3d tracking and fore-casting with graph neural network and diversity sampling. InarXiv:2003.07847, 2020. 2

[34] X. Weng Y. Wang, K. Kitani. Joint object detectionand multi-object tracking with graph neural networks. InarXiv:2006.13164 [cs.CV], 2020. 2

[35] Xiaohui Zeng, Renjie Liao, Li Gu, Yuwen Xiong, Sanja Fi-dler, and Raquel Urtasun. Dmm-net: Differentiable mask-matching network for video object segmentation. In Pro-ceedings of the IEEE International Conference on ComputerVision, pages 3929–3938, 2019. 2

[36] Wenwei Zhang, Hui Zhou, Shuyang Sun, Zhe Wang, Jian-ping Shi, and Chen Change Loy. Robust multi-modalitymulti-object tracking. In International Conference on Com-puter Vision (ICCV), October 2019. 9

[37] Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenjun Zeng,and Wenyu Liu. Fairmot: On the fairness of detectionand re-identification in multiple object tracking. arXiv:2004.01888, 2020. 2

[38] Xingyi Zhou, Vladlen Koltun, and Philipp Krahenbuhl.Tracking objects as points. ECCV, 2020. 2, 9