Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

h

Animal Tracking for Behavior Modeling

James M. RehgGeorgia Tech

(standing in for Tucker Balch)

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hPeople

• PI Tucker Balch

• Faculty Jim Rehg, Computer Science

− Collaborator on animal tracking Aaron Bobick, Computer Science

− Collaborator on animal behavior modeling Bruce Walker, Psychology

− Collaborator on aquarium project Atsushi Nakazawa, Computer Science (Osaka Univ.,

Japan)− Collaborator on animal tracking

• PhD Students Michael Novitzky Jin Lee Matthew Flagg

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hGoals

•Track social animals reliably under natural conditions (in vivo) from video

•Estimate behavioral parameters from tracking data

•Construct executable models of social animal behavior

•Develop a biologically-inspired protocol for dynamic team formation

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hYear 1 Overview

•Goal Develop reliable multi-target tracking

algorithms for animals in video•Approach

Jointly estimate segmentation and motion of a nonrigid, deformable target

•Key requirements Reliably estimate target shape over time, to

support behavioral analysis Minimize the amount of human effort required

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hChallenges

•Accurate segmentation of target Set-point tracking is not sufficient

•Track reliably with significant camera motion Background subtraction is not sufficient

•Handle a wide range of animals Scalable solution for the animal kingdom

•Track multiple interacting targets Multiple instances of target type (e.g. wolf pack) Occlusions (with self, other targets, and

background)•Track long sequences, track across cuts

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hApproach

•Basic research in video object segmentation and tracking

•Modular software architecture Easily change features, models, and solver

•Testing in two scenarios Aquarium Monitoring

−Long sequences, static camera, some modeling effort Tracking Animal Planet videos

−Shorter sequences, moving camera, minimal human effort

•Preliminary application Accessible Aquarium Project

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hTracking App

Basic Trackers: -Contour feature w/ Iterated Closest Point-Color feature w/ Mean Shift

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hColor Histogram Tracker

• Fish Model: Appearance model For each species of fish,

multiple HSV Color Histograms on Image patches

Off-line Model Selection by Human

• Detection and Tracking: Histogram-based mean shift

approach Maximization of Bhattacharyya

Coefficient between Observation and Model

Selection of a model with the highest coefficient and update track

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hHistogram-based Model

Selection

Model Image Patches(to build histogram)Input Image Similarity

between shifted &

model patches

0.60.50.70.80.3 Best Model

*

Image Patch after Mean-shifting

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hAccessible Aquarium

Project•Provide a meaningful and informative aquarium

experience for visually-impaired or blind visitors•Approach

Track the movement of individual fish within tank Sonify the fish movement

•Example: 65 gallon marine aquarium Track yellow tangs and blue chromis Music structure is Bach chorale Each fish type is same instrument, different registers Movement speed mapped to note density (tempo) Horizontal dimension is stereo, vertical is timbre

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hExample

•Play video

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hTracking in the Video

Volume

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hGraphcut Tracking

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hTracking Results

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hSocial Game Retrieval

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hOverview

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hSummary of Progress

•Robust long-life tracking of multiple targets (fish) Tracking under controlled (but realistic)

conditions•Virtual Aquarium Project

Novel experience of animal behavior via sonification

•New state-of-art motion segmentation algorithm Accurate segmentation of (shorter) video

sequences under a wide range conditions

Robo

tics a

nd In

tellig

ent M

achi

nes @

 Geo

rgia

Tec

hYear 2 Plans

•Make general tracking method work for longer sequences

•Develop integrated tracking application tool for biologists

•Connect to biologists more effectively (what to say about this?)

•Start developing behavior models, how?