ETISEO, Nice, May 11-12 2005

PETSPETSInternational Workshops onInternational Workshops on

Performance Evaluation of Tracking and SurveillancePerformance Evaluation of Tracking and Surveillance

James FerrymanJames FerrymanComputational Vision GroupComputational Vision Group

Department of Computer ScienceDepartment of Computer ScienceThe University of Reading, UKThe University of Reading, UK

ETISEO, Nice, May 11-12 2005

Supported by

PETSPETSInternational Workshops onInternational Workshops on

Performance Evaluation of Tracking and SurveillancePerformance Evaluation of Tracking and Surveillance

ETISEO, Nice, May 11-12 2005

IntroductionIntroduction Theme - Performance Evaluation of

Tracking and Surveillance Successful tracking of object motions key

to visual surveillance PETS started in Grenoble, France in 2000

as satellite workshop of FG2000 Not a competition http://visualsurveillance.org ftp://pets.rdg.ac.uk

ETISEO, Nice, May 11-12 2005

PETS - HistoryPETS - History PETS’2000 was held at FG’2000, 31 March 2000,

Grenoble, France. PETS’2001 at CVPR’01. PETS’2002 at ECCV, Copenhagen, Denmark, June

1 2002. PETS2003 at ICVS, Graz; VS-PETS at ICCV2003 PETS2004 at ECCV04 WAMOP-PETS, CO, USA (Jan 05) as part of IEEE

Winter Workshop Series 2005: VS-PETS at ICCV’05

ETISEO, Nice, May 11-12 2005

Datasets – Example – PETS2001Datasets – Example – PETS2001

Five separate sets of training and test sequences.

All datasets are multiview (frame sychronised).

Datasets were significantly more challenging than PETS2000 (significant lighting variation, occlusion, scene activity and use of multiview data)

ETISEO, Nice, May 11-12 2005

DatasetsDatasets

Dataset 2

Dataset 1

Dataset 3

ETISEO, Nice, May 11-12 2005

Dataset 1Dataset 1

ETISEO, Nice, May 11-12 2005

Dataset 2Dataset 2

ETISEO, Nice, May 11-12 2005

Dataset 4Dataset 4

ETISEO, Nice, May 11-12 2005

PETS - PrerequisitesPETS - Prerequisites

Tracking results reported– should be performed using the test sequences, but the

training sequences may optionally be used if the algorithms require it (for learning etc.)

– may be based on a single camera view of the scene, or using multiple view data.

– can be based on the entire test sequence, or a portion of it; the images may be converted to any other format and/or subsampled.

– results must be submitted in XML format.

ETISEO, Nice, May 11-12 2005

PETS – Workshop OverviewPETS – Workshop Overview

XX contributed papers~3 sessions: e.g. appearance-based

tracking, people and vehicle tracking, multiview tracking

Y invited speakersDemonstration sessionOverall evaluation and discussion

ETISEO, Nice, May 11-12 2005

Quantitative PE - XMLQuantitative PE - XML

XML provides mechanism of setting up “syntax” file in form of schema

Schema used to automatically validate object tracking results

For PETS’2001, two schemas were used:– low-level tracking results– high-level surveillance

(understanding object motions and interactions)

ETISEO, Nice, May 11-12 2005

Quantitative PE - XMLQuantitative PE - XML<?xml version='1.0' encoding='ISO-8859-1' ?><!-- --><!-- Example file for visual surveillance reporting --><!-- -> scene understanding with multiple cameras. -->

<!-- Edited by PETS2001.Reading.JMF (J.M.Ferryman@reading.ac.uk) -->

<people_tracker xmlns="http://www.cvg.cs.reading.ac.uk/PETS2001" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation= "http://www.cvg.cs.reading.ac.uk/PETS2001

http://www.cvg.cs.reading.ac.uk/PETS2001/XML/surveillance.xsd">

<!-- this is a comment ... add more as appropriate -->

<header> <recording site="PETS2001 (Reading)" session="1" date="01/06/01"> <list_cameras num_cameras="2">

<camera camera_id="1"/> <camera camera_id="2"/> </list_cameras> </recording>

ETISEO, Nice, May 11-12 2005

Quantitative PE - XMLQuantitative PE - XML<video start_frame="1" end_frame="1450" step="2" fps="25"/> <!-- step: stepping to read list of processed frames below --> <!-- (for (i=start; i<=end; i+=step); i++) --> <!-- fps: frames per second of original video --> <image xdim="768" ydim="576" colour="1"/> <!-- dimensions of video images and whether it is in colour (0,1) -->

<software name="Reading People Tracker" platform="Linux" version="0.03" run_date="12/07/00"> <!-- information about the software this file originates from -->

<object_detector name="Reading People Tracker" platform="Linux" version="0.03" run_date="08/06/01"/>

</software>

</header>

Quantitative PE - XMLQuantitative PE - XML<sequence> <!-- the actual data: a sequence of one or more frames. -->

<frame id="2" num_targets="1">

<target id="8" start_frame="1" end_frame ="2"> <!-- a "target" is any object which moves or may move, usually a person, group of people, or a vehicle. The target's id is GLOBAL to all the cameras defined in "list_cameras" -->

<!-- start_frame and end_frame indicate when the target has been tracked. end_frame may be unknown because it is in the future; in this case the longest known time where the object was tracked will be given -->

<track status="4" location="0" speed="200" trajectory="1" t_confidence="0.5" num_parents="2">

<!-- track is part of a graph representing tracks of all targets --> <!-- the status of a graph node explains how the node of current target has been created or tracked. The following values may be used and added together as appropriate: 0 : default value, already tracked 1 : new track (id did not exist before) 2 : re-appearing object (id copied from last occurrence) 4 : merging (more than one parent in graph) 8 : splitting (at least one parent in graph has more than 1 child) 16 : lost (object NOT found in current image, given position etc are estimates (if available) or previous values) 32 : out of field of view (tracked object not "visible" as per definition (see elsewhere)) -->

<!-- location values are defined as the sum of the following:

0 : undefined 1 : roadway 2 : in close proximity to vehicle (parking lot) 4 : on grass/verge 8 : other -->

ETISEO, Nice, May 11-12 2005

D1C1: XML outputD1C1: XML output

ETISEO, Nice, May 11-12 2005

D1 C1 - 1

ETISEO, Nice, May 11-12 2005

D1 C1 - 1

ETISEO, Nice, May 11-12 2005

D1 C1 - 1

ETISEO, Nice, May 11-12 2005

D1 C1 - 1

ETISEO, Nice, May 11-12 2005

D1 C1 - 1

ETISEO, Nice, May 11-12 2005

D1 C1 - 1

ETISEO, Nice, May 11-12 2005

D1 C1 - 2

ETISEO, Nice, May 11-12 2005

D1 C1 - 2

ETISEO, Nice, May 11-12 2005

D1 C1 - 2

ETISEO, Nice, May 11-12 2005

D1 C1 - 2

ETISEO, Nice, May 11-12 2005

D1 C1 - 2

ETISEO, Nice, May 11-12 2005

D1 C1- 3

ETISEO, Nice, May 11-12 2005

D1 C1- 3

ETISEO, Nice, May 11-12 2005

D1 C1- 3

ETISEO, Nice, May 11-12 2005

D1 C1- 3

ETISEO, Nice, May 11-12 2005

D1 C1- 3

ETISEO, Nice, May 11-12 2005

D1 C1- 3

ETISEO, Nice, May 11-12 2005

D1C1: XML output 1D1C1: XML output 1

ETISEO, Nice, May 11-12 2005

D1C1: XML output 2D1C1: XML output 2

ETISEO, Nice, May 11-12 2005

D1C1: XML output 3D1C1: XML output 3

ETISEO, Nice, May 11-12 2005

D1C1: XML output 4D1C1: XML output 4

ETISEO, Nice, May 11-12 2005

D1C1: XML output 5D1C1: XML output 5

ETISEO, Nice, May 11-12 2005

Performance EvaluationPerformance Evaluation

Evaluation of surveillance system can be judged in a number of ways:– object detection lag– object centroid position error– object area error– track incompleteness factor– accuracy of semantics of interaction– object identity error

• maintenance of identity through occlusion• …

ETISEO, Nice, May 11-12 2005

384 x 288 (JPEG)

10 fps

384 x 288 25-30 fps 600 MHzDual PIII850 MHz

384 x 288 (AVI)384 x 288 5 fps768 x 576 5 fps384 x 288 6.25 fps 800 MHz PIII320 x 240 29.97 fps 1 GHz PIV

5fps 1.7 GHz PIV768 X 576 5fps

Image Format Processing Speed Processor

ETISEO, Nice, May 11-12 2005

DiscussionDiscussion Evaluation criteria are application

dependent Training data – required or not?

representative exampleshow much?

Semantics of XML schema Ground truth

difficult to obtainautomatic evaluation - how?

ETISEO, Nice, May 11-12 2005

PETS EvaluationPETS Evaluation +ve: “Mindset” – engaging the community –

change of culture +ve: Repository of data (PETS01 most frequently

accessed) +ve: Discussion/presentation of methodologies,

metrics, tools … +ve: Filters through to conferences/published

literature -ve: For workshop, choice of dataset(s)

+ annotation -ve: More quantitative evaluation needed

ETISEO, Nice, May 11-12 2005

PETS, ETISEO and the future …PETS, ETISEO and the future … Online web-based evaluation service

(Semi-)automatic validation of XML against ground truth

Repository of algorithms (incl. “strawman”), and tabulated results (rank?)

Methodology for evaluationMetrics

More challenging datasets (e.g. multiview) Live workshop sessions on “unseen” data Expectation that ETISEO will support PETS

ETISEO, Nice, May 11-12 2005

PETS’05PETS’05

ICCV ’05, Beijing, China15-16 October 2005http://www.cbsr.ia.ac.cn/

conferences/VS-PETS-2005 http://visualsurveillance.org/PETS2005