SIViP (2017) 11:1287–1295DOI 10.1007/s11760-017-1086-7

ORIGINAL PAPER

A method for performance diagnosis and evaluation of videotrackers

Tahir Nawaz1 · Anna Ellis1 · James Ferryman1

Received: 22 December 2016 / Revised: 19 February 2017 / Accepted: 20 March 2017 / Published online: 30 March 2017© The Author(s) 2017. This article is an open access publication

Abstract Several measures for evaluating multi-targetvideo trackers exist that generally aim at providing ‘endperformance.’ End performance is important particularlyfor ranking and comparing trackers. However, for a deeperinsight into trackers’ performance it would also be desirableto analyze key contributory factors (false positives, false neg-atives, ID changes) that (implicitly or explicitly) lead to theattainment of a certain end performance. Specifically, thispaper proposes a new approach to enable a diagnosis of theperformance of multi-target trackers as well as providing ameans to determine the end performance to still enable theircomparison in a video sequence. Diagnosis involves ana-lyzing probability density functions of false positives, falsenegatives and ID changes of trackers in a sequence. Endperformance is obtained in terms of the extracted perfor-mance scores related to false positives, false negatives andID changes. In the experiments, we used four state-of-the-arttrackers on challenging real-world public datasets to showthe effectiveness of the proposed approach.

Keywords Video tracking · Performance diagnosis ·Performance evaluation

This work has received funding from the European Union’s SeventhFramework Programme for research, technological development anddemonstration under Grant Agreement No. 312784.

B Tahir Nawazt.h.nawaz@reading.ac.uk

Anna Ellisa.l.ellis@reading.ac.uk

James Ferrymanj.m.ferryman@reading.ac.uk

1 Computational Vision Group, Department of ComputerScience, University of Reading, Reading, UK

1 Introduction

Evaluation measures [4–6,9,15,19,22] are important tech-niques of providing a means to draw performance com-parisons among different multi-target tracking algorithms[3,17,18,20,21]. These measures are generally aimed todetermine end performance of trackers. End performanceprovides an overall quantification of goodness or badness oftrackers’ results in the form of a score at frame level [12,19],or sequence level [4,15], without separately analyzing in anexplicit manner the key factors (i.e., false positives, false neg-atives, ID changes [4]) that contribute to the achievement ofa certain performance score. Analysis of these contributoryfactors may indeed be needed in interpreting performancebehavior of tracking algorithms against a variety of datasets.It would therefore be desirable from a researcher’s perspec-tive to obtain a deeper insight into these factors in additionto the end performance.

Existingmeasures are broadlymade up of composite errorcounts [4,15], tracking success counts [5,12], tracking fail-ure counts [11,14] and temporal averaging of scores [13,15];providing tracking quality measurements without giving anexplicit insight as why the performance of a tracker is lessthan perfect. Consider, for example, two cases where a well-known existingmeasure,MultipleObject TrackingAccuracy(MOTA) [4], provides a comparable performance for a pair ofmulti-target trackers on a dataset, and ranks one tracker to bebetter than the other on another dataset (Fig. 1a). Indeed,the measure provides an end performance comparison inthe form of a score for each tracker but does not reveal aswhy those end performances are obtained by trackers. Thereappears to be a need to also perform a diagnosis that is aimedat revealing and dissecting different aspects of tracking per-formance that could help in understanding why a certainperformance is achieved. Such an approach may be used

123

1288 SIViP (2017) 11:1287–1295

ETH Sunnyday dataset iLids Easy dataset0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ATO

M

PoiesiTrackerPirsiavashTracker

Comparable end performance!

Different end performance!

(b)(a)

Fig. 1 a In given examples, MOTA provides a comparable perfor-mance for a tracker by Poiesi et al. (PoiesiTracker) [18] and a trackerby Pirsiavash et al. (PirsiavashTracker) [17] on ETH Sunnyday dataset[8], and ranks latter better than former on iLids Easy dataset [7]. b Def-inition of a false positive (FP), a false negative (FN) and an ID change(IDC) in a frame. Ground truth: dotted bounding box; tracker’s result:solid bounding box. Bounding box color represents a unique ID

in conjunction with end performance measures to provide aclearer and a more detailed picture of tracker’s performance.Next we formally define the problem to further clarify whatdiagnosis means in this paper.

1.1 Problem definition

Let X be a set of tracks estimated by a multi-target trackerin a video sequence, V : X = {X j }Jj=1, where J is the totalnumber of estimated tracks.X j is the estimated track for tar-

get j : X j = (Xk, j )k jend

k=k jstart

, where k jstart and k j

end are the first

and final frame numbers of X j , respectively. Xk, j is the esti-mated state of target j at frame k : k = 1, . . . , K with K asthe total number of frames inV . Xk, j = (xk, j , yk, j , Ak, j , l j ),where (xk, j , yk, j ) and Ak, j denote at frame k the positionand occupied area of target j on image plane, respectively,and l j defines its ID. Ak, j may use rectangular (boundingbox) [18], elliptical [10], or contour [1] representations. Thenumber of estimated targets at frame k is denoted as nk ,which are defined as {Xk,1, . . . , Xk, j , . . . , Xk,nk }. Likewise,the notations for the ground-truth quantities correspondingtoX ,X j , J, Xk, j , k

jstart, k

jend, xk, j , yk, j , Ak, j , l j and nk are

X , Xi , I, Xk,i , kistart, kiend, xk,i , yk,i , Ak,i , li and nk , respec-

tively.A typical diagnostic procedure for a system starts as a

result of identification of symptom(s) that may allude tothe deterioration in system’s performance. For a multi-targettracking system, deterioration may refer to deviation of Xfrom X [16], which is computed as a discrepancy betweenX and X . The deterioration of performance in a systemresults from the occurence of fault(s) in it. In the case ofa tracking system, the basic set of faults may include IDchange, false positive, and false negative; referring to theerror in maintaining a unique target ID, incorrect estima-tion, and missed estimation at frame k, respectively (see Fig.1b). Here, we consider these three frame-level faults (IDchange, false positive, false negative) as they often implicitlyor explicitly form a basis for, or contribute to, estimat-

ing existing track-level assessment proposals [4–6,15,19,22](see details in Sect. 2). A diagnostic procedure involvesperforming fault diagnosis. Therefore, for a tracking sys-tem, diagnosis may include analyzing across the frames ofa sequence the occurrence of false positives, false negatives,and ID changes, which is expected to dissect and reveal moreinto the achievement of a certain end performance.

1.2 Contributions

In this paper, we present a new approach that, instead of pre-senting only the end performance of trackers, is also aimedat diagnosis in terms of providing a more revealing pic-ture of the performance of multi-target trackers. It involvesanalyzing probability density functions (PDFs), in additionto extracting performance scores for each fault type (falsepositives, false negatives, ID changes) in a video sequence.Performance scores quantify the per frame concentration androbustness of a tracker to each fault type.We show the useful-ness of the proposed method using state-of-the-art trackerson four challenging publicly available datasets.

2 Related work

Measures exist that implicitly account for faults (falsepositives, false negatives, ID changes) in their formula-tion to provide end tracking performance, with respect tothe ground-truth information [15,19]. Optimal Sub-PatternAssignment (OSPA)metric [19] provides a frame-level targetpositional evaluation by combining accuracy and cardinalityerrors. The cardinality error (difference between the num-ber of estimated and ground-truth targets) is the numberof unassociated targets at frame k; hence, it encapsulatesthe information about false positives and false negatives.Inspired from OSPA, Multiple Extended-target TrackingError (METE) [15] also quantifies frame-level performanceby combining accuracy and cardinality errors; taking alsointo account the information about the occupied targetregion. Multiple Extended-target Lost-Track ratio (MELT)[15] quantifies the performance at sequence level basedon the use of lost-track ratio. Given an associated pair ofestimated and ground-truth tracks, lost-track ratio is com-puted as a ratio of the number of frames with an overlap

(O( Ak,i , Ak, j ) = | Ak,i∩Ak, j || Ak,i∪Ak, j | , such that |.| is the cardinality

of a set) between a pair of Xk, j and Xk,i less than a predefinedthreshold value, and the number of frames in a ground-truthtrack i . When O(·) is less than the threshold, it may pointtoward the presence of a false positive, or a false negative ina frame.

Some measures explicitly use information about the faultand combine them to quantify the end performance [2,4,5].

123

SIViP (2017) 11:1287–1295 1289

False Alarm Rate (FAR) [2,5], Specificity, Positive Predic-tion, and False Positive Rate [2] use the number of falsepositives with other quantities in the evaluation procedureat frame level. Negative Prediction and False Negative Rate[2] use information about the number of false negatives withother quantities in evaluation at frame level. Multiple ObjectTracking Accuracy (MOTA) [4] estimates performance bycombining information about the number of false positives,false negatives, and ID changes at each frame and normaliz-ing across the sequence.

Measures that quantify performance by separately usingthe information from a specific fault include NormalizedID changes (NIDC) [15], False Positive track matches andFalse Negative track matches [6], False Alarm Track (FAT)and Track Detection Failure (TDF) [22], Track Fragmenta-tion (TF) [5], and ID Changes (IDC) [22]. NIDC normalizesthe number of ID changes by length of the track in whichthey occur. False Positive track matches and FAT use infor-mation about false positives across frames. False Negativetrack matches and TDF use information about false nega-tives across frames. TF and IDC count the number of IDchanges across frames of an individual track and all tracks,respectively.

As reviewed above, existing measures (OSPA, METE,MELT, MOTA, FAR, Specificity, Positive and NegativePredictions, False Positive and Negative Rates) focus onevaluating end performance of trackers without separatelyproviding an explicit insight into each fault type that couldbe needed to understand the attainment of a certain end per-formance. Somemeasures (NIDC, FAT, TDF, TF, IDC, FalsePositive and False Negative track matches) do provide a sep-arate evaluation for each fault type; however, counting (orcombining) false positives, false negatives, or ID changeswould still provide an end performance evaluation that maynot enable understanding tracker’s performance behavior interms of its ability to deal with each fault type. In this paper,we address this limitation by proposing an approach thatinvolves dissecting a tracker’s performance by separatelyanalyzing the behavior of each fault type, while still enablingthe end performance evaluation.

3 Tracking performance diagnosis and evaluation

Without loss of generality, Ak, j is considered in the formof a bounding box in which case Xk, j can be re-writtenas: Xk, j = (xk, j , yk, j , wk, j , hk, j , l j ), where wk, j and hk, jdenote width and height of the bounding box for target jat frame k. The notations for ground-truth quantities corre-sponding to wk, j and hk, j are wk,i and hk,i , respectively.Given a set of estimated states {Xk,1, . . . , Xk, j , . . . , Xk,nk },and a set of ground-truth states {Xk,1, . . . , Xk,i , . . . , Xk,nk }at frame k, the association between the elements of the two

sets is established using Hungarian algorithm by minimizingthe overlap cost

(1 − O(·)), where O(·) defines the amount

of overlap between a pair of Xk, j and Xk,i , as described in

Sect. 2, i.e., O( Ak,i , Ak, j ) = | Ak,i∩Ak, j || Ak,i∪Ak, j | . FPk , the false pos-

itives, are the number of associated pairs of estimated andground-truth targets with O(·) < τ (where τ is a thresholdvalue) plus the number of unassociated estimated targets atframe k. FNk , the false negatives, are the number of ground-truth targets that are missed by a tracker at frame k. IDCk ,the ID changes, are the number of changed associations cor-responding to the ground-truth tracks at frame k. See alsoFig. 1b. Next, we describe the proposed method includingperformance diagnosis (Sect. 3.1) and evaluation (Sect. 3.2),followed by highlighting the advantages of using the pro-posed method.

3.1 Performance diagnosis

Analyzing the occurence of FPk , FNk , and IDCk at eachframe can be cumbersome for longer sequences, and alsomake it difficult to analyze and compare trackers’ perfor-mance across different sequences with different lengths.Additionally, as discussed in the Sect. 2, looking solely atthe total numbers of false positives, false negatives, or IDchanges across a sequence is still an end performance evalua-tion, and a deeper insight would be desirable for performancediagnosis. Instead, the analysis of the distributions of falsepositives, false negatives, and ID changes in a sequence isexpected to provide a more revealing picture of tracker’s per-formance behavior for a fault type, irrespective of sequencelength. Moreover, the analysis of distributions (in normal-ized form) could enable inferring trends about performanceof trackers across different datasets.

We therefore compute probability density functions(PDFs) for false positives, false negatives, and ID changes ina sequence. A PDF is computed as a normalized histogramfor a particular fault type; hence, the area under each PDFequals 1, i.e., the sum of bin values on y-axis is equal to 1.We denote PDFs for false positives, false negatives, and IDchanges for a sequence as Pr [0 ≤ FPk ≤ Nfp], Pr [0 ≤FNk ≤ Nfn] and Pr [0 ≤ IDCk ≤ Nidc], respectively, whereNfp = max{FPk}Kk=1, Nfn = max{FNk}Kk=1, and Nidc =max{IDCk}Kk=1. Figure 2 shows PDFs of false positives, falsenegatives, and ID changes for existing trackers with thedatasets used in this study. For example: Pr [FPk = 0] isread as a probability in terms of the percentage of framesin which the tracker produces zero false positive; similarly,Pr [FNk = 2] refers to a probability in terms of the per-centage of frames in which the tracker produces two falsenegatives; likewise, Pr [IDCk > 2] means a probability interms of the percentage of frames in which the tracker pro-duces more than two ID changes.

123

1290 SIViP (2017) 11:1287–1295

Towncentre

0 1 2 3 4 5 6 7 8 9 10 11 120

0.2

0.4

0.6

0.8

1

FPk

BenfoldTracker-HPoiesiTracker-HBenfoldTrackerPirsiavashTracker

0 1 2 3 4 5 6 7 8 9 10 11 120

0.2

0.4

0.6

0.8

1

Pro

babi

lity

dens

ity

FNk

BenfoldTracker-HPoiesiTracker-HBenfoldTrackerPirsiavashTracker

0 1 2 3 4 5 6 7 8 90

0.2

0.4

0.6

0.8

1

Pro

babi

lity

dens

ityP

roba

bilit

y de

nsity

IDCk

BenfoldTracker-HPoiesiTracker-HBenfoldTrackerPirsiavashTracker

(c)(b)(a)

iLids Easy

0 1 2 3 40

0.2

0.4

0.6

0.8

1

FPk

BenfoldTrackerPoiesiTrackerPirsiavashTracker

0 1 2 3 40

0.2

0.4

0.6

0.8

1

Pro

babi

lity

dens

ity

Pro

babi

lity

dens

ityP

roba

bilit

y de

nsity

Pro

babi

lity

dens

ity

Pro

babi

lity

dens

ityP

roba

bilit

y de

nsity

Pro

babi

lity

dens

ityP

roba

bilit

y de

nsity

Pro

babi

lity

dens

ity

FNk

BenfoldTrackerPoiesiTrackerPirsiavashTracker

0 1 2 30

0.2

0.4

0.6

0.8

1

IDCk

BenfoldTrackerPoiesiTrackerPirsiavashTracker

(f)(e)(d)

ETH Bahnhof

0 1 2 3 4 5 60

0.2

0.4

0.6

0.8

1

FPk

YangTrackerPoiesiTrackerPirsiavashTracker

0 1 2 3 4 5 6 7 8 9 10 11 120

0.2

0.4

0.6

0.8

1

FNk

YangTrackerPoiesiTrackerPirsiavashTracker

0 1 2 3 4 50

0.2

0.4

0.6

0.8

1

IDCk

YangTrackerPoiesiTrackerPirsiavashTracker

(i)(h)(g)

ETH Sunnyday

0 1 2 3 4 50

0.2

0.4

0.6

0.8

1

FPk

YangTrackerPoiesiTrackerPirsiavashTracker

0 1 2 3 4 50

0.2

0.4

0.6

0.8

1

FNk

YangTrackerPoiesiTrackerPirsiavashTracker

0 1 2 30

0.2

0.4

0.6

0.8

1

IDCk

YangTrackerPoiesiTrackerPirsiavashTracker

(l)(k)(j)

Fig. 2 Performance diagnosis in terms of the probability density func-tions (PDFs) of trackers on the Towncentre [3], iLids Easy [7], ETHBahnhof [8], and ETH Sunnyday [8] datasets for fault types: false

positives (first column); false negatives (second column); ID changes(third column). On Towncentre, the legends, ‘BenfoldTracker-H’ and‘PoiesiTracker-H,’ refer to the use of these trackers for head tracking

To show the usefulness of analyzing PDFs, consider thecase of the Towncentre dataset [3], where PDFs for false pos-itives of a pair of trackers (a tracker from Benfold and Reid(BenfoldTracker) [3] and a tracker from Pirsiavash et al. (Pir-siavashTracker) [17]) are generated for full-body tracking inFig. 2a (PDFs are shown with solid lines). The total numberof false positives in the sequence (

∑Kk=1 FPk) only reveals

that PirsiavashTracker (10,118 false positives) is better thanBenfoldTracker (12,162 false positives); however, their cor-responding PDFs (Fig. 2a) provide a deeper insight asfollows. PDFs reveal that Pr [FPk = 0] for BenfoldTracker ishigher (better) than Pr [FPk = 0] for PirsiavashTracker. Thisshows an enhanced robustness of BenfoldTracker than Pirsi-avashTracker because of the presence of more frames wherethe former did not produce any false positives. The PDFsfurther reveal that, on the contrary, BenfoldTracker showsa greater tendency than PirsiavashTracker of producing ahigher concentration of false positives (i.e., for FPk > 3) in

a frame (see Fig. 2a), i.e., Pr [FPk > 3] for BenfoldTrackeris higher (worse) than that for PirsiavashTracker. Therefore,analysis of a PDF offers amore detailed and dissected pictureof a tracker’s performance by revealing its robustness and perframe concentration for an individual fault type, which is notexplicitly available by a simple fault count. To further aid theanalysis and to facilitate end performance evaluation com-parison of trackers, we next define two performance scoresthat account for the two aspects above for each fault type.

3.2 Performance evaluation

The first score tells the ability of a tracker to track withoutproducing a fault across a sequence, and is called robustnessto a fault type (R): Rfp = 1 − Kfp

K ; Rfn = 1 − KfnK ; Ridc =

1 − KidcK ; such that Kfp is the number of frames containing

false positive(s), Kfn is the number of frames containing false

123

SIViP (2017) 11:1287–1295 1291

negative(s), and Kidc is the number of frames containing IDchange(s). Rfp ∈ [0, 1], Rfn ∈ [0, 1], Ridc ∈ [0, 1]: thehigher the value (Rfp/Rfn/Ridc), the better the ability.NB: Rfp/Rfn/Ridc differs in formulation from MOTA [4]:

MOTA = 1 −∑K

k (c1FPk+c2FNk+c3IDCk )∑Kk=1 nk

. If c1 = 1, c2 =0, c3 = 0, or c1 = 0, c2 = 1, c3 = 0, or c1 = 0, c2 =0, c3 = 1, MOTA reduces to providing the performance sep-arately in terms of false positives or false negatives or IDchanges, respectively. UnlikeMOTA that, in ‘reduced’ form,uses information about the number of false positives, falsenegatives, or ID changes, Rfp, Rfn and Ridc instead use infor-mation about the number of frames having false positive(s),false negative(s), and ID change(s), respectively, to providerobustness.

The second score tells the tendency of a tracker toproduce a fault type per frame, and is called per frame con-centration of a fault type (PFC): PFCfp = 1

K

∑Kk=1 FPk ,

PFCfn = 1K

∑Kk=1 FNk , and PFCidc = 1

K

∑Kk=1 IDCk .

PFCfp ≥ 0,PFCfn ≥ 0, and PFCidc ≥ 0: the lower the value(PFCfp/PFCfn/PFCidc), the lower the tendency of producinga fault type per frame.

NB: PFCidc differs from NIDC [15], such that the latterpenalizes the number of ID changes by length of the track inwhich they occur, whereas the former quantifies per frameconcentration by averaging the number of ID changes acrossthe whole sequence. Likewise, PFCfp and PFCfn differ fromMELT [15] that encapsulates lost-track ratio information (asexplained inSect. 2). The lost-track value could indeed reflectthe number of frames having false positives and/or false neg-atives in a track. Unlike MELT, PFCfp(PFCfn) quantifies perframe concentration by averaging the number of false posi-tives(false negatives) across the whole sequence.

3.3 Advantages

This section shows the advantages of using the proposedmethod over the widely used measure, MOTA. To this end,for clarity, we plot in Fig. 3 the numerator term of MOTA(that we here refer to as MOTAk : MOTAk = ∑K

k (c1FPk +c2FNk+c3IDCk)) for BenfoldTracker [3] on a segment of theTowncentre dataset. MOTAk combines contributions of FPk ,

1300 1310 1320 1330 1340 1350 1360 1370 1380 1390 14000

2

4

6

8

10

12

k

Mea

sure

MOTAk

FPk

FNk

IDCk

Fig. 3 The numerator term of MOTA (here referred to as MOTAk ) isplotted across a segment of the Towncentre dataset for BenfoldTracker.Additionally, FPk ,FNk and IDCk are also plotted alongside

FNk , and IDCk at frame k. Indeed the same value of MOTAk

could be caused by different combinations of FPk , FNk , andIDCk ; for example, MOTAk = 8 at k = 1348 and k = 1393,although values of FPk , FNk , and IDCk are different in theseframes (see Fig. 3). Therefore, MOTA alone might not berevealing enough, as it does not provide an explicit insightinto the individual fault types (false positives, false negatives,ID changes) that could be beneficial for a deeper understand-ingof performance.Differently, the proposedmethod enablesa separate analysis of the behavior of individual fault typesfor a tracker in terms of respective PDFs (Sect. 3.1), as wellas its end performance in terms of extracted robustness (R)and per frame concentration (PFC) scores for each fault type(Sect. 3.2).

4 Experimental validation

This section demonstrates the usefulness of the proposedmethod using state-of-the-art trackers on real-world publiclyavailable datasets. Section 4.1 describes the setup includingtrackers and datasets, followed by the performance analysisof trackers using the proposed method and (for comparison)existing measures in Sect. 4.2, and a discussion in Sect. 4.3.

4.1 Trackers and datasets

Table 1 provides a summary of trackers and datasets usedin the experiments. We used available ground truth gener-ated for every frame of the sequences. We used trackersfrom Pirsiavash et al. (PirsiavashTracker) [17], Yang andNevatia (YangTracker) [21], Benfold and Reid (Benfold-Tracker) [3], and Poiesi et al. (PoiesiTracker) [18]. Theparameters of trackers are the same as in the original papers.We use head and full-body tracks in experiments. More-over, we chose four challenging datasets: Towncentre [3],iLids Easy [7], ETH Bahnhof [8], and ETH Sunnyday[8]. Towncentre and iLids Easy are recorded from a staticcamera, whereas ETH Bahnhof and Sunnyday involves amoving camera. On Towncenter, trackers are tested for headtracking (BenfoldTracker-H, PoiesiTracker-H) and full-bodytracking (BenfoldTracker, PirsiavashTracker); on iLids Easy,trackers are used for full-body tracking (BenfoldTracker,PoiesiTracker, PirsiavashTracker); and on ETHBahnhof andSunnyday, trackers are tested for full-body tracking (Yang-Tracker, PoiesiTracker, PirsiavashTracker).We use τ = 0.25for head tracking, and τ = 0.5 for full-body tracking [3].Figure 2 shows PDFs of trackers for each fault type on alldatasets. Table 2 presents performance scores (PFCfp, PFCfn,PFCidc, Rfp, Rfn, Ridc) and existing measures (MOTA, meanMETE, MELT) for trackers on all datasets, as well as theirnumber of false positives, false negatives, and ID changes toaid analysis.

123

1292 SIViP (2017) 11:1287–1295

Table 1 Summary of datasetsDataset K Frame size I Challenges Trackers TT

Towncentre 4491 1080 × 1920 231 Occ, SC, Cr [3,17,18] H, P

iLids Easy 5220 576 × 720 17 Occ, SC, IC [3,17,18] P

ETH Bahnhof 998 480 × 640 95 Occ, SC, Cr, IC, CM [17,18,21] P

ETH Sunnyday 353 480 × 640 30 Occ, SC, Cr, IC, CM [17,18,21] P

K , number of frames; I , number of trajectories; TT, target type(s) under consideration; H, ‘head’ target; P,‘full person body’ target; Occ, occlusion; SC, scale changes; Cr, crowdedness; IC, illumination changes; CM,camera motion

4.2 Performance analysis of trackers

4.2.1 Towncentre

For full-body tracking, existing measures (MOTA, meanMETE, MELT) are primarily limited to only showing thatBenfoldTracker outperforms PirsiavashTracker (Table 2).Differently, the proposed method provides a greater under-standing of the trackers’ performance, by enabling a separateanalysis of their per frame concentration and robustness tofaults using corresponding PDFs, as well as PFC and Rscores, as follows. Based on false positives, PDFs of Ben-foldTracker and PirsiavashTracker (shown with solid linesin Fig. 2a) reveal that there are more frames in whichPirsiavashTracker produced false positive(s) than Benfold-Tracker. This shows that the latter is more robust to falsepositives than the former, which is also confirmed by a betterRfp for BenfoldTracker (Table 2); see also qualitative resultswhere PirsiavashTracker produced false positives, but Ben-foldTracker did not (Fig. 4b). At the same time, it can alsobe noticed that BenfoldTracker shows a greater tendency ofproducing a higher per frame concentration of false posi-tives than PirsiavashTracker (i.e., for FPk > 3, Fig. 2a),that is also shown by a better PFCfp for PirsiavashTracker(Table 2). Based on false negatives and ID changes, Benfold-Tracker outperforms PirsiavashTracker, as shown in generalin their PDFs (Fig. 2b, c), PFCfn, PFCidc, Rfn, and Ridc (Table2).

Likewise, for head tracking on this dataset, the val-ues of MOTA, mean METE and MELT simply show thatPoiesiTracker-H is better than BenfoldTracker-H (Table2). The proposed method enables a more detailed anal-ysis of trackers’ performance as follows. PDFs of track-ers (BenfoldTracker-H, PoiesiTracker-H) are shown withdotted lines in Fig. 2a–c. For false positives (Fig. 2a),the results show that Pr [FPk = 0] for PoiesiTracker-H is higher than that for BenfoldTracker-H, showing anenhanced robustness of the former to false positives thatis also noticeable by its superior Rfp (Table 2). As for perframe concentration, from PDFs, there is no clear winnerbetween BenfoldTracker-H and PoiesiTracker-H for FPk >

0 (Fig. 2a): BenfoldTracker-H outperforms PoiesiTracker-

H for 0 < FPk ≤ 3, PoiesiTracker-H is better thanBenfoldTracker-H for 3 < FPk < 7, and both trackersgenerally perform comparably thereafter across their PDFs.Overall, in terms of PFCfp, PoiesiTracker-H, however, showsa superior performance than BenfoldTracker-H (Table 2).For false negatives, PoiesiTracker-H shows more robust-ness than BenfoldTracker-H, that is noticeable by higherPr [FNk = 0] (Fig. 2b) and Rfn (Table 2) of former.As for per frame concentration, BenfoldTracker-H showsa better PFCfn than PoiesiTracker-H (Table 2); this is alsoreflected bymostly a better performance of the former acrosstheir PDFs, i.e., for FNk > 3 (Fig. 2b). See qualitativeresults in a sample frame showing several false negativesfor PoiesiTracker-H, and a fewer for BenfoldTracker-H (Fig.4c). For ID changes, overall the results based on PDFs (Fig.2c) and PFCidc, Ridc (Table 2) reveal that PoiesiTracker-His better based on per frame concentration of ID changes,whereas BenfoldTracker-H is more robust to producing IDchanges.

4.2.2 iLids Easy

Based on MOTA, mean METE and MELT, BenfoldTrackeris the best followed by PirsiavashTracker and PoiesiTracker(Table 2). The proposed method produces a different rank-ing based on false positives (PFCfp, Rfp) and ID changes(PFCidc, Ridc) by ranking PoiesiTracker as the best, fol-lowed by PirsiavashTracker and BenfoldTracker (Table 2).Indeed, Fig. 2d, f also shows that PoiesiTracker generallyoutperforms PirsiavashTracker and BenfoldTracker acrosstheir PDFs of false positives and ID changes. On the otherhand, based on false negatives, the performance trends oftrackers using the proposed method are similar to thoseproduced by MOTA, mean METE and MELT, i.e., Benfold-Tracker outperforms PirsiavashTracker and PoiesiTrackeracross their PDFs (Fig. 2e), as well as based on theirPFCfn, Rfn (Table 2). The qualitative results show thatBenfoldTracker produces more false positives (Fig. 4d,f) and ID changes (Fig. 4d, e) than others, and Poiesi-Tracker produces more false negatives (Fig. 4d) than oth-ers.

123

SIViP (2017) 11:1287–1295 1293

Tabl

e2

Performance

evaluatio

nof

trackerson

differentd

atasetsusingPF

Cfp

,PF

Cfn

,PF

Cidc,Rfp

,Rfn,and

Ridc

Tracker

Dataset

PFCfp

PFCfn

PFCidc

Rfp

Rfn

Ridc

FPFN

IDC

MOTA

MeanMETE

MELT

BenfoldTracker-H

Towncentre

3.836

1.19

20.426

0.029

0.526

0.80

417

,228

5354

1913

0.678

0.622

0.656

PoiesiTracker-H

3.36

81.283

0.40

00.

050

0.54

60.796

15,1

255762

1798

0.70

10.

534

0.55

7

BenfoldTracker

2.708

1.19

00.

338

0.07

20.

530

0.81

712

,162

5345

1519

0.75

00.

329

0.38

6

PirsiavashTracker

2.25

32.418

0.587

0.046

0.253

0.650

10,1

1810

,861

2637

0.697

0.479

0.542

BenfoldTracker

iLidsEasy

0.458

0.17

70.043

0.679

0.85

00.959

2392

926

227

0.74

30.

356

0.42

8

PoiesiTracker

0.16

40.939

0.01

00.

852

0.472

0.99

285

84900

540.562

0.530

0.541

PirsiavashTracker

0.301

0.440

0.020

0.805

0.652

0.981

1571

2296

105

0.704

0.404

0.523

YangT

racker

ETHBahnhof

0.826

1.264

0.15

80.427

0.388

0.88

3824

1262

158

0.74

40.

394

0.41

9

PoiesiTracker

1.426

1.11

90.308

0.292

0.47

20.796

1423

1117

307

0.685

0.443

0.461

PirsiavashTracker

0.23

43.065

0.229

0.79

20.073

0.818

234

3059

229

0.597

0.529

0.568

YangT

racker

ETHSu

nnyday

0.705

0.821

0.08

80.490

0.414

0.91

5249

290

310.

711

0.460

0.39

0

PoiesiTracker

0.940

0.66

00.159

0.428

0.55

50.881

332

233

560.694

0.468

0.461

PirsiavashTracker

0.43

91.153

0.122

0.62

60.346

0.887

155

407

430.697

0.44

40.560

The

totalnum

beroffalsepositiv

es(FP),falsenegativ

es(FN),ID

changes(ID

C),MOTA

,MeanMETEandMELT

oftrackersarealso

listed.Fo

rTow

ncentre,‘B

enfoldTracker-H

’and

‘PoiesiTracker-H

’referto

theuseof

thesetrackersforhead

tracking.O

neach

dataset,thebesttracking

scores

areshow

nin

bold

4.2.3 ETH Bahnhof and Sunnyday

On ETH Bahnhof, existing measures (MOTA, mean METE,MELT) consider YangTracker as the best, followed byPoiesiTracker and PirsiavashTracker. The proposed methodprovides additional information and useful insights into theperformance based on different fault types as follows. Basedon false positives, PirsiavashTracker is found to be the mostrobust and shows the least per frame concentration, fol-lowed by YangTracker and PoiesiTracker, as confirmed bytheir PFCfp and Rfp scores (Table 2), and across their PDFs(Fig. 2g). For example, Fig. 4g, i shows the qualitativetracking results with more false positives for PoiesiTrackerthan others. Based on false negatives, PoiesiTracker is thebest, followed by YangTracker and PirsiavashTracker; thisis reflected by their PFCfn and Rfn (Table 2), as well asin general across their PDFs (Fig. 2h). See, for example,qualitative results in a sample frame with more false nega-tives for PrisiavashTracker than others (Fig. 4i). Based on IDchanges, YangTracker shows an increased robustness and abetter per frame concentration, as compared to Pirsiavash-Tracker and PoiesiTracker, as confirmed by their PFCidc andRidc scores (Table 2), and generally across their PDFs (Fig.2i). On ETH Sunnyday, the trends and rankings of track-ers (YangTracker, PirsiavashTracker, PoiesiTracker) basedon PFC and R scores (Table 2), and PDFs (Fig. 2j–l) for allfault types are interestingly similar to those reported abovefor ETH Bahnhof. See also qualitative results on ETH Sun-nyday in Fig. 4j–l.

4.3 Discussion

The proposed method could be used to provide formativefeedback that could help researchers in addressing shortcom-ings in tracking algorithms. In fact, the analysis based onfalse positives could enable analyzing the impact on track-ing performance originating from the detection stage. Forexample,BenfoldTracker has generally shown inferior PFCfp

and Rfp on Towncentre and iLids Easy than others, andPoiesiTracker has shown inferior PFCfp and Rfp on ETHBahnhof and Sunnyday than others. Indeed, on ETH Bahn-hof and Sunnyday, the possible reason of the worst PFCfp

and Rfp scores of PoiesiTracker is that its person detectorhas a limited ability to deal with varying illumination con-ditions in these datasets [15]. Therefore, the results showa particular need of improvement at the detection stage ofBenfoldTracker and PoiesiTracker. Similarly, inferior scoresrelated to false negatives (i.e., PFCfn, Rfn) can point towardimproving the detection stage, and/or inability to tempo-rally link small tracks (‘tracklets’) in an effective manner.For example, PirsiavashTracker has shown inferior PFCfn

and Rfn on most datasets (Towncentre, ETH Bahnhof, ETHSunnyday) than others. This is likely due to the absence

123

1294 SIViP (2017) 11:1287–1295

(a) (b) (c)

(d) (e) (f)

(g) (h) (i)

(j) (k) (l)

Fig. 4 Qualitative results of trackers on the Towncentre [3] (a–c), iLids Easy [7] (d–f), ETH Bahnhof [8] (g–i), and ETH Sunnyday [8] (j–l)datasets. Key—Red BenfoldTracker, blue PirsiavashTracker, green PoiesiTracker, black YangTracker

of an effective dedicated strategy to link tracklets in Pirsi-avashTracker [17], which other trackers (e.g., PoiesiTracker,YangTracker) possess. In fact, it is due to this limited abilitythat PirsiavashTracker also reported the highest cardinal-ity error (that can be caused by false negatives) on thesedatasets in an earlier study [15]. Likewise, the analysis basedon ID changes provide a formative feedback vis-a-vis thetracking stage. For instance, YangTracker consistently showsbetter PFCidc and Ridc than PoiesiTracker and Pirsiavash-Tracker onETHBahnhof and Sunnyday,which also confirmsthe conclusions of [15] that YangTracker outperforms othertrackers in terms of ID changes on the same datasets. Indeed,this is because YangTracker uses an effective ID manage-ment strategy, employing motion and appearance affinitiesto avoid confusion between IDs of targets that are close toeach other [21]. Hence, a researcher could pay more atten-tion on improving the ID management in PoiesiTracker andPirsiavashTracker.

5 Conclusions

We presented a new method that, instead of just providingusual end performance evaluation, also aims at performance

diagnosis of a multi-target tracker in a video sequence. Exist-ing tracking evaluation proposals generally focus only on endperformance assessment that is important for drawing perfor-mance comparison. Instead, the proposed approach enables amore detailed performance analysis using probability densityfunctions (PDFs) of key frame-level faults that a tracker canmake (i.e., false positives, false negatives, ID changes). Tocomplement this analysis, the extracted performance scoresfurther offer a separate evaluation in terms of per frame con-centration and robustness of trackers for each fault type. Weused real-world publicly available datasets using state-of-the-art trackers to validate the proposed method by showingits effectiveness over existing proposals, and its use in iden-tifying algorithmic shortcomings of trackers.

While the proposed method accounts for multi-targettracking, it could still be partly suitable for single-targettrackers; however, for single-target tracking, ID change isgenerally not an issue and, hence, could be ignored. Addi-tionally, the proposed method could be applied for any targettype, provided the target model contains the position andoccupied area (on a 2D image plane) as parameters. More-over, the proposed method is based on analyzing frame-level

123

SIViP (2017) 11:1287–1295 1295

faults. An explicit inclusion of track-level faults could alsobe of interest and is left to future work.

Open Access This article is distributed under the terms of the CreativeCommons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution,and reproduction in any medium, provided you give appropriate creditto the original author(s) and the source, provide a link to the CreativeCommons license, and indicate if changes were made.

References

1. Allili, M.S., Ziou, D.: Active contours for video object trackingusing region, boundary and shape information. SIVP 1(2), 101–117 (2007)

2. Bashir, F., Porikli, F.: Performance evaluation of object detectionand tracking systems. In: Proceedings of IEEE PETS Workshop(2006)

3. Benfold, B., Reid, I.: Stable multi-target tracking in real-timesurveillance video. In: Proceedings of IEEE Conference on CVPR,Colorado Springs, USA (2011)

4. Bernardin,K., Stiefelhagen,R.: Evaluatingmultiple object trackingperformance: the clear mot metrics. JIVP 2008, 1–10 (2008)

5. Black, J., Ellis, T., Rosin, P.: A novel method for video trackingperformance evaluation. In: Proceedings of IEEE PETSWorkshop(2003)

6. Brown, L.M., Senior, A.W., Tian, Y.L., Connell, J., Hampapur, A.,Shu, C.F., Merkl, H., Lu, M.: Performance evaluation of surveil-lance systems under varying conditions. In: Proceedings of IEEEPETS Workshop (2005)

7. http://www.eecs.qmul.ac.uk/~andrea/avss2007_d.html. AccessedJan 2016

8. http://www.vision.ee.ethz.ch/~aess/iccv2007/. Accessed Jan 20169. Kasturi, R., Goldgof, D., Soundararajan, P.,Manohar, V., Garofolo,

J., Bowers, R., Boonstra, M., Korzhova, V., Zhang, J.: Frameworkfor performance evaluation of face, text, and vehicle detection andtracking in video: data, metrics, and protocol. IEEE Trans. PAMI31(2), 319–336 (2009)

10. Koohzadi, M., Keyvanpour, M.: OTWC: an efficient object-tracking method. SIVP 9(6), 1235–1247 (2015)

11. Kristan, M., Pflugfelder, R., Leonardis, A., Matas, J., Porikli, F.,Cehovin, L., Nebehay, G., Fernandez, G., Vojir, T.: The VOT2013challenge: overview and additional results. In: CVWW (2014)

12. Li, H., Shen, C., Shi, Q.: Real-time visual tracking using com-pressive sensing. In: Proceedings of IEEE Conference on CVPR(2011)

13. Nawaz, T., Cavallaro, A.: PFT: a protocol for evaluating videotrackers. In: Proceedings of ICIP, Brussels (2011)

14. Nawaz, T., Cavallaro, A.: A protocol for evaluating video trackersunder real-world conditions. IEEE Trans. Image Process. 22(4),1354–1361 (2013)

15. Nawaz, T., Poiesi, F., Cavallaro, A.: Measures of effective videotracking. IEEE Trans. Image Process. 23(1), 376–388 (2014)

16. Nawaz, T.H.: Ground-truth-based trajectory evaluation in videos.Ph.D. thesis, Queen Mary University of London, UK (2014)

17. Pirsiavash, H., Ramanan, D., Fowlkes, C.C.: Globally-optimalgreedy algorithms for tracking avariable number of objects. In: Pro-ceedings of IEEE Conference on CVPR, Colorado Springs, USA(2011)

18. Poiesi, F., Mazzon, R., Cavallaro, A.: Multi-target tracking onconfidence maps: an application to people tracking. Comput. Vis.Image Underst. 117(10), 1257–1272 (2013)

19. Ristic, B., Vo, B.N., Clark, D., Vo, B.T.: A metric for performanceevaluation of multi-target tracking algorithms. IEEE Trans. SignalProcess. 59(7), 3452–3457 (2011)

20. Wu, H., Liu, N., Luo, X., Su, J., Chen, L.: Real-time backgroundsubtraction-based video surveillance of people by integrating localtexture patterns. SIVP 8(4), 665–676 (2014)

21. Yang, B., Nevatia, R.: An online learned CRF model for multi-target tracking. In: Proceedings of IEEE Conference on CVPR,Providence, Rhode Island (2012)

22. Yin, F.,Makris,D.,Velastin, S.A.: Performance evaluationof objecttracking algorithms. In: Proceedings of IEEE PETS Workshop(2007)

123