Chaotic Invariants for Human Action Recognition

Saad Ali, Arslan Basharat, Mubarak ShahICCV 2007

Outline

• Related Work• Chaotic Systems Preliminaries

– Dynamical Systems, Phase Portrait, Chaos, Orbits etc.

• Proposed Action Representation– Phase Space Embedding

• Features (Chaotic Invariants)– Lyapunov Exponent, Correlation Integral, Correlation Dimension,

• Experimental Results

Actions in the Computer Vision Literature

• Categorization on the basis of representation

Geometrical Models of Human Body PartsMori et. al 2004, Jiang et. al 2006

Space Time Pattern TemplatesBobick et. al 1996, Irani et. al 2005

Volumetric Feature - Yilmaz et. al 2005, Shechtman et. al. 2005, Mahmood et. al. 01

Space Time Interest Points Laptev et. al. 2005, Fei-Fei et. al. 2006

Motion/Optical Flow Patterns Yacoob et. al 1999, Efros et. al 2003.

Actions in the Computer Vision Literature

– Linear Dynamical Models – Bissacco et. al. CVPR 2001.

• Trajectories are output of a linear dynamical system driven by white Gaussian noise

– Non-Linear Dynamical Models• Bregler CVPR 1997 – Used finite automata to

switch between linear dynamical models.• Ralaivola et. al. NIPS 2004 – Proposed kernels to

extend linear dynamical models.• Wang et. al. NIPS 2005 – Gaussian process

dynamical models to handle non-linear time series data.

Actions in the Computer Vision Literature

• Only approximation – Constrains the dynamical model to be of a particular type.

• Trying to fit experimental data to the model.

• Not letting experimental/observed data to decide the type of the dynamical system.

• Analogous Example– Gaussian PDF vs Kernel Density Estimation

• Overcome these limitations by using concepts from the Chaos Theory.

Preliminaries

• Dynamical System – A set of states along with a rule(function) that describe how present state is determined by previous state.

• Dynamical systems within them have a determinism i.e., characterized by a deterministic function.

• Deterministic Behavior – Only depend on previous state + initial condition

Preliminaries cont’d

An Example of a Dynamical System

Petri Dish

Bacteria

State:

=xPopulation of bacteria at time ‘t’Initial Conditions:

10000 =xA rule describing the evolution of the state:

tt xx 21 =+

Preliminaries• Map – A function which is a set to itself i.e. domain = range

• Fixed Point – f(p) = p

• Phase Space – Space of state variables (domain of the deterministic function)

• Orbit – Path traced by evolution of system’s states

x y

Domain

0x)( 0xf ))(( 0xff )))((( 0xfff

Preliminaries• Phase Portrait – Collection of all orbits from all

possible starting points in the phase space.

• Attractor – Region in phase space to which all orbits settle down.

• Strange Attractor – If attractor is not stable.

• Invariants of Attractor – Measures that are invariant under smooth transformation of phase space.– Metric– Dynamical– Topological

Preliminaries• Metric – Measures structure of the phase space using metric

measurements.

• Topological – Depend on periodic orbits of attractor.

• Dynamical – Based on behavior of orbits over time.

• Embedding – Mapping of one dimensional signal to m-dimensional signal.

• Chaos/Chaotic Dynamics: A descriptive term for the properties of a dynamical system.

• Chaos Theory: Study of apparently random behavior of deterministic systems.

• Chaos only occurs in non-linear systems.

PreliminariesAn Example of a Chaotic System

Ed Lorenz found :

-20 -10 0 10 20 30

-20

0

20

0

10

20

30

40

50 σ=10, r=28, b=8/3

X

Lorenz System

Y

Z xy - bz z rx-y-xz y

x)y σ x

==

−=

&

&

& ( The solution is called a strange attractor.-It oscillates irregularly, never exactly repeating.

Proposed Idea

),...,,( 332

31

+++ tN

ttf θθθ),...,,( 21tN

ttf θθθ ),...,,( 112

11

+++ tN

ttf θθθ ),...,,( 222

21

+++ tN

ttf θθθ

f ),...,,( 21tN

tt θθθ

Rule True State Space VariablesUnknown We have the access to the data generatedby the dynamical system controlling this action !

Proposed Idea

• Experimental Data: Trajectories of body joints

• From this data construct the phase space (and get periodic strange attractors) corresponding to the dynamical system responsible for generating the data.

• That is: Let the data speak to you and tell you what mechanisms are generating chaotic behavior.

Algorithmic OverviewAction Video Time Series: X,Y

Convert into time seriesExtract Trajectories

Joint Trajectories

- 2 - 1 . 5 - 1 - 0 . 5 0 0 . 5 1 1 . 5- 1 8

- 1 6

- 1 4

- 1 2

- 1 0

-8

-6

l n r

ln C

(r)

C o r r e la t io n s u m

- 5 - 4 - 3 - 2 - 1 0- 9

- 8

- 7

- 6

- 5

- 4

- 3

- 2

- 1

0

ld r

ld C

(r)

S c a l in g o f D 2

0 5 0 1 0 0 1 5 0 2 0 00

0 . 5

1

1 . 5

2

p

P r e d i c t io n e r r o r

- 2 - 1 . 5 - 1 - 0 . 5 0 0 . 5 1 1 . 5- 1 8

- 1 6

- 1 4

- 1 2

- 1 0

- 8

- 6

l n r

ln C

(r)

C o r r e la t io n s u m

- 5 - 4 - 3 - 2 - 1 0- 9

- 8

- 7

- 6

- 5

- 4

- 3

- 2

- 1

0

ld r

ld C

(r)

S c a l in g o f D 2

0 5 0 1 0 0 1 5 0 2 0 00

0 . 5

1

1 . 5

2

p

P r e d i c t io n e r r o r

- 2 - 1 . 5 -1 - 0 . 5 0 0 . 5 1 1 . 5- 1 8

- 1 6

- 1 4

- 1 2

- 1 0

-8

-6

l n r

ln C

(r)

C o r r e la t io n s u m

- 5 - 4 - 3 - 2 - 1 0- 9

- 8

- 7

- 6

- 5

- 4

- 3

- 2

- 1

0

ld r

ld C

(r)

S c a l in g o f D 2

0 5 0 1 0 0 1 5 0 2 0 00

0 . 5

1

1 . 5

2

p

P r e d i c t io n e r r o r

For each time series perform Phase Space Embedding

…………..-20

020

40

-50

0

50-20

-10

0

10

20

30

Reconstructed PhaseSpace

-200

2040

-50

0

50-20

-10

0

10

20

30

Reconstructed PhaseSpace

-200

2040

-20

0

20

40-20

-10

0

10

20

30

Reconstructed PhaseSpace

Compute Chaotic Invariants

…………..

-2 -1.5 -1 -0 .5 0 0 .5 1 1 .5-18

-16

-14

-12

-10

-8

-6

ln r

ln C

(r)

Co rre lation sum

-5 -4 -3 -2 -1 0-9

-8

-7

-6

-5

-4

-3

-2

-1

0

ld r

ld C

(r)

Scaling of D 2

0 50 100 150 2000

0.5

1

1.5

2

p

Prediction error

LyapunovExponent

CorrelationIntegral

CorrelationDimension

Chaotic Invariants

-2 -1.5 -1 -0 .5 0 0 .5 1 1 .5-18

-16

-14

-12

-10

-8

-6

ln r

ln C

(r)

Co rre lation sum

-5 -4 -3 -2 -1 0-9

-8

-7

-6

-5

-4

-3

-2

-1

0

ld r

ld C

(r)

Scaling of D 2

0 50 100 150 2000

0.5

1

1.5

2

p

Prediction error

LyapunovExponent

CorrelationIntegral

CorrelationDimension

Chaotic Invariants

-2 -1.5 -1 -0 .5 0 0 .5 1 1 .5-18

-16

-14

-12

-10

-8

-6

ln r

ln C

(r)

Co rre lation sum

-5 -4 -3 -2 -1 0-9

-8

-7

-6

-5

-4

-3

-2

-1

0

ld r

ld C

(r)

Scaling of D 2

0 50 100 150 2000

0.5

1

1.5

2

p

Prediction error

LyapunovExponent

CorrelationIntegral

CorrelationDimension

Chaotic Invariants

Action Representation

• Six Body Joints– Two Hands, Two Feet, Head, Belly.

• Normalized with respect to the belly point.

• Results in 5 trajectories per action.

Action Representation

• Each dimension of the trajectory is considered as a univariate time series

Phase Space Embedding• Underlying Idea: All the variables of the dynamical

system influence each other.

tzzzz ,....,,, 321=τ

Therefore, can be considered as a second

substitute variable which carries the influence of all the systems variables during time interval .

τ+iz

τ

Every point of the series results from the intricate

combination of influences of all the true state variables.

),...,,( 21 Nθθθ

iz

τττ miii zzz +++ ,....,, 32

Using this reasoning, introduce a series of substitute variables and obtain the whole m-dimensional space.

Phase Space Embedding• Taken’s Theorem: States that a map exist between original space

and reconstructed space.

• Thus for optimal and , delay vectors

generates a phase space that has exactly the same properties as the original/true variables of the system.

• Substitute variables - Carry the same information as the original

mττττ )1(2 ....,,, −+++ miiii zzzz

variables only with respect to the properties associated with the system’s dynamics.

• Posses no additional or particular physical meaning.

Embedding Delay

• How to find optimal and ?

• For use mutual information between &

• Mutual information between and quantifies the amount of information we have about the state assuming we know

m τ

τzτ+z

iz

iz τ+iz

τ+iz

tzzzzzzzz ,....,,,,,,, 7654321

τ τ

Embedding Delay

• Properties of optimal– Large enough so that value at is significantly

different from what we already know at time

– Should not be larger than the typical time in which system losses its memory.

• First minima of the mutual information as the optimal embedding delay.

ττ+i

i

Embedding Delay – The Algorithm

tzzzz ,....,,, 321Input:

1. Compute

2. Divide d into j equal size bins.

3. Compute

maxmin zzd −=

∑ ∑= =

−=j

h

j

k kh

khkj PP

PPI

1 1

,,

)(ln)()(

τττ

hP Probability that variable assumes a value inside hth bin

kP Probability that variable assumes a value inside kth bin

τ+izkhP , Probability that is in hth bin and is in kth biniz

Embedding Delay

Embedding Dimension

• False Nearest Neighbor Algorithm.

• Unfold the observed orbits from self overlap arising from projection of system’s attractor to a lower dimensional space.

[ ]21 , xx

[ ]21 , yy

[ ]1x [ ]1y

=d

Embedding Dimension - The Algorithm

• Given p(i) in a m-dimensional space find a neighbor p(j) so that:

• Calculate normalized distance between (m+1)th coordinates of p(i) and p(j)

• If larger than a threshold, p(i) is marked as having a false nearest neighbor.

• Repeat for various values of m until fraction of points for which is negligible.

iR

ξ<− )()( jpip

thresholdRi >

iR)()( jpip

xxR mjmi

i −

−= ++ ττ

Embedding Dimension

Reconstructed Phase Space

tzzzzzzzz ,....,,,,,,, 7654321

τ τ

3=m2=τ

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

...642

531

zzzzzz

X Each row is a point in a m-dimensional

phase space.

m-dimensional reconstructed phase space

Phase Spaces

Head

Right H

and

Left Hand

Right Foot

Left Foot

Invariant Features

• Maximal Lyapunov Exponent

• Correlation Integral

• Correlation Dimension

Maximal Lyapunov Exponent

• Dynamical Invariant

• Measures exponential divergence of nearby trajectories in phase space.

• Maximal Lyapunov exponent > 0, implies dynamics of the underlying system are chaotic.

Maximal Lyapunov Exponent – The Algorithm

• Pick a reference point p(i) in the phase space

• Find all points p(k) within a neighborhood radius

• Compute average distance of all trajectories to reference trajectory as a function of relative time

‘s’ counts different points p(k)

∑=

∆+−+∆+−+ −=∆r

snminmki zz

rnD

1)1()1(

1)( τ

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

=−++

−++

−

...........

)1(222

)1(111

)1(0

ττ

ττ

ττ

m

m

m

zzzzzzzzz

X

ξ

‘r’ total points

Maximal Lyapunov Exponent – The Algorithm

• Repeat process for several reference points and take average of the log distance

• Maximum Lyapunov component is slope of the line fitted to the curve of to

∑=

∆=∆c

ii nD

cnS

1))(ln(1)(

)( nS ∆ n∆

Maximal Lyapunov Exponent

Correlation Integral• Metric Invariant

• Measures the number of points within a neighborhood of radius , averaged over the entire attractor

))()(()1(*

2)(1 1

jpipNN

CN

i

N

ij−−Θ

−= ∑ ∑

= +=

εε

ξ

Correlation Dimension

• Characterizes metric structure of the attractor.

• Measures the change in the density of phase space with respect to neighborhood radius.

• Computation proceeds by plotting correlation integral against neighborhood radius.

Correlation Dimension

Experiment-I• Motion capture data

• Dataset size– Dance : 19– Run : 26 – Walk : 46– Sit: 14– Jump: 33

• Leave-One-Out Cross validation using K-means classifier.

Experiments Motion Capture DataWalking Running Jumping

Ballet Sitting

Experiments Motion Capture Data

Dance

Run

Experiments Motion Capture Data

Walk

Jump Sit

Experiments Motion Capture Data

Dance Jump Run Sit Walk

Dance 28 2

Jump 13 1

Run 2 1 22 1 4

Sit 33

Walk 3 2 43

Mean Accuracy: 89.7%

Experiment -II

• Wizemann Action Data Set

• Nine actions performed by nine different actors:– Bend, Jumping Jack, Jump Forward, Jump in

Place, Run, Side Gallop, Walk, Wave One Hand, Wave Two Hands

• 81 videos

Experiment-II

Experiments Wizemann Action Data Set

A1 A2 A3 A4 A5 A6 A7 A8 A9A1 9A2 9A3 5 2 2A4 9A5 8 1A6 1 8A7 9A8 9A9 9

Mean Accuracy: 92.6%

A1: Bend, A2: Jumping Jack, A3: Jump in Place, A4: Run, A5: Side GallopA6: Walk, A7: Wave1, A8: Wave2

Thank You