Recognition of Human Recognition of Human GaitsGaits

Alessandro BissaccoAlessandro BissaccoComputer ScienceComputer Science

UCLAUCLA

Yi MaYi MaElectrical and Computer Electrical and Computer

EngineeringEngineeringUniversity of Illinois at Urbana-University of Illinois at Urbana-

ChampaignChampaign

Alessandro ChiusoAlessandro ChiusoDipartimento di Elettronica e Dipartimento di Elettronica e

InformaticaInformaticaUniversita di PadovaUniversita di Padova

Stefano SoattoStefano SoattoComputer ScienceComputer Science

UCLAUCLA

Presented by Peter Schwer

SummarySummary

Problem DescriptionProblem Description Related WorkRelated Work Overview of Gait Recognition ProcessOverview of Gait Recognition Process

TrackingTracking Learning the ModelLearning the Model RecognitionRecognition

ResultsResults

The Human GaitThe Human Gait

Observable from great distancesObservable from great distances Discernable independent of skew (relative Discernable independent of skew (relative

direction of subject’s gait)direction of subject’s gait) Communicates:Communicates:

IdentityIdentity MoodMood

Good Recognition TraitsGood Recognition Traits

Invariant to:Invariant to: Photometric Factors:Photometric Factors:

Illumination, clothingIllumination, clothing Geometric Factors:Geometric Factors:

Camera PositionCamera PositionLength of subject’s limbsLength of subject’s limbsRelative direction of gaitRelative direction of gait

Good Recognition TraitsGood Recognition Traits

Capable of discerning:Capable of discerning: Different types of gaits:Different types of gaits:

RunningRunningWalkingWalkingSkipping, etc.Skipping, etc.

MoodMoodNot addressed, but dynamical models Not addressed, but dynamical models

can be extended to this area.can be extended to this area.

Related WorkRelated Work

Analyzing gait with spatiotemporal surfaces Analyzing gait with spatiotemporal surfaces (1994) Sourabh A. Niyogi, Edward H. (1994) Sourabh A. Niyogi, Edward H. AdelsonAdelson

University of SouthamptonCarnegie Mellon University (CMU)Massachusetts Insitute of Technology (MIT)Georgia Tech (GATech)University of MarylandUniversity of South Florida

The Recognition ProblemThe Recognition Problem

The Sub-problems:The Sub-problems:TrackingTrackingLearning the modelLearning the modelRecognizingRecognizing

TrackingTracking

““The emphasis of our work is The emphasis of our work is notnot tracking.” tracking.” Images Images → Skeletons→ Skeletons

Results:Results:

y(t), t = 1 … y(t), t = 1 … ττ

(Sequence of joint positions)(Sequence of joint positions)

Learning the ModelLearning the Model

The Right ModelThe Right Model

Ideally,Ideally, Unique model for each datasetUnique model for each dataset Method for calculating similarity (distance) between Method for calculating similarity (distance) between

modelsmodels

Catch:Catch: ““The set of dynamical models in canonical form is The set of dynamical models in canonical form is notnot

a linear space (even if the model itself is linear!)”a linear space (even if the model itself is linear!)” Distances between dynamical models are not trivially Distances between dynamical models are not trivially

computed.computed. Data, as usual, is inherently noisy.Data, as usual, is inherently noisy.

Creating the Dynamical ModelCreating the Dynamical Model

Input: Input: y(t)y(t) – Sequence of Joint Positions – Sequence of Joint Positions What are some of the statistical properties of What are some of the statistical properties of

this sequence of joint positions?this sequence of joint positions? Stochastic Stationary Second-order processStochastic Stationary Second-order process How is noise distributed in the data?How is noise distributed in the data?

We use a Gauss-Markov Model as a We use a Gauss-Markov Model as a representative of each class.representative of each class.

The ARMA ModelThe ARMA Modelx(t+1) = Ax(t) + v(t), v(t) ~ N(0,Q); x(0) = x0x(t+1) = Ax(t) + v(t), v(t) ~ N(0,Q); x(0) = x0y(t) = C(x(t) + w(t); w(t) ~ N(0,R)y(t) = C(x(t) + w(t); w(t) ~ N(0,R)

The ARMA ModelThe ARMA Model

Other Applications:Other Applications: Speech RecognitionSpeech Recognition TrackingTracking Geophysics: Forecasting and SimulationGeophysics: Forecasting and Simulation Pattern RecognitionPattern Recognition Dynamic Texture RecognitionDynamic Texture Recognition

The ARMA modelThe ARMA model

x(t) is a process describing the “state” at time t (vector)x(t) is a process describing the “state” at time t (vector)y(t) is the measured joint positions (vector)y(t) is the measured joint positions (vector)Q,R are input and output noise covariances (matrices)Q,R are input and output noise covariances (matrices)

x(t+1) = Ax(t) + v(t), v(t) ~ x(t+1) = Ax(t) + v(t), v(t) ~ NN(0,Q); x(0) = x(0,Q); x(0) = x00

y(t) = C(x(t) + w(t); w(t) ~ y(t) = C(x(t) + w(t); w(t) ~ NN(0,R)(0,R)

((AA:[n :[n ×× n], n], B B:[m :[m × n])× n])

PROBLEM: Non-uniqueness of A, C, Q, R, S

Innovation/Canonical Innovation/Canonical RepresentationRepresentation

Identify stochastic subspace models Identify stochastic subspace models without forming the covariance matrix.without forming the covariance matrix.

Principal Angles and DirectionsPrincipal Angles and Directions Source:Source:

P. Van Overschee and B. De MoorP. Van Overschee and B. De Moor

Subspace Algorithms for the stochastic Subspace Algorithms for the stochastic identification problem. identification problem. AutomaticaAutomatica, 29:649-, 29:649-660, 1993.660, 1993.

A Canonical RepresentationA Canonical Representation

x(t) is a process describing the “state” (vector)x(t) is a process describing the “state” (vector)y(t) is the measured joint positions (vector)y(t) is the measured joint positions (vector)Q,R are input and output noise covariances (matrices)Q,R are input and output noise covariances (matrices)

x(t+1) = Ax(t) + v(t), v(t) ~ x(t+1) = Ax(t) + v(t), v(t) ~ NN(0,Q); x(0) = x(0,Q); x(0) = x00

y(t) = C(x(t) + w(t); w(t) ~ y(t) = C(x(t) + w(t); w(t) ~ NN(0,R)(0,R)

From Overschee and De Moor we have {A,C,Q,R}From Overschee and De Moor we have {A,C,Q,R}

A and C describe the gait.A and C describe the gait.

RecognizingRecognizing

Our Dynamic ModelOur Dynamic Model

A, C, Q, R providedA, C, Q, R provided What are they?What are they? How might they be useful?How might they be useful?

How to Describe GaitHow to Describe Gait

Tracking produces a “waveform” that describes Tracking produces a “waveform” that describes a series of joint positions/angles:a series of joint positions/angles:

Gait 1

Gait 2

What is the best way to compare

these to waveforms?

Distance Between ModelsDistance Between Models

Subspace/principal anglesSubspace/principal angles Start with:Start with: O(M)- infinite observability matrix O(M)- infinite observability matrix

0 ),,...,,(

:tuple-n by the denoted are and subspaces thebetween angles principal The

)(cos),...,(cos :are

matrix theof aluessingular v ordered-n The

ization)orthogonalSchmidt -Gramby (Obtain

. as subspace same thespans that basis lorthonorma thebe Let

subspace. ldimensiona- an spanning columns hmatrix wita be Let

)(

12121

21

21

2

2

21

iin

n

nnH

TH

H

H

nTTTTT

HH

HH

QQ

Q

HQ

nH

CACACM

O

Form Form O(m)O(m)

O(m)O(m) is a subspace is a subspace We want to compare the observability We want to compare the observability

subspaces of two modelssubspaces of two models MM11 and and MM22..

Form Form O(m)O(m) for each model: for each model:

nTTTTT CACACM 2)(O

Call the results Call the results HH11 and and HH22, , respectively.respectively.

Form Orthonormal MatricesForm Orthonormal Matrices

Gram-Schmidt Orthogonalization on Gram-Schmidt Orthogonalization on HH11 and and HH22 Described: Described:

http://lagrange.la.asu.edu/VirtualClass/Algebra/Gramhttp://lagrange.la.asu.edu/VirtualClass/Algebra/GramSchm.htmlSchm.html

Applet: Applet: http://http://www.mste.uiuc.edu/exner/ncsa/orthogonal/#simulatiowww.mste.uiuc.edu/exner/ncsa/orthogonal/#simulationn

Call the result orthonormal matrices Call the result orthonormal matrices QQH1H1 and and QQH2H2

Calculate Principal AnglesCalculate Principal Angles

Compute n-ordered Singular Values of Compute n-ordered Singular Values of QQTT

H1H1 QQH2H2

n-ordered singular values can be denoted: n-ordered singular values can be denoted: coscos22((θθ11), … , ), … , coscos22((θθnn))

Principal Angles between the subspaces Principal Angles between the subspaces HH11 and and HH22 are denoted by the n-tuple: are denoted by the n-tuple: ((θθ1 1 , , θθ22 ,…, ,…, θθn n ) sorted in descending order.) sorted in descending order.

Calculate the DistanceCalculate the Distance First measure is not necessarily non-negative (less fit)First measure is not necessarily non-negative (less fit) Second Measure is the largest principal angle (more Second Measure is the largest principal angle (more

fit)fit)

1

222

:Norm) (

)(cosln :Norm) (

F

iiM

dL

or

dL

The Martin and Finsler Distance, respectively.The Martin and Finsler Distance, respectively.

OrOr

Just use their matlab code:Just use their matlab code:function theta = subspace_angles(A1,K1,C1,A2,K2,C2)n = size(A1,1);m = size(C1,1);A = [ A1 zeros(n,3*n); zeros(n) A2-K2*C2 zeros(n,2*n);zeros(n,2*n) A2 zeros(n); zeros(n,3*n) A1-K1*C1];C = [ C1 -C2 C2 -C1 ];Q = dlyap(A’,C’*C);E = eig([zeros(2*n) pinv(Q(1:2*n,1:2*n))*Q(1:2*n,2*n+1:4*n);pinv(Q(2*n+1:4*n,2*n+1:4*n))*Q(2*n+1:4*n,1:2*n)zeros(2*n)]);E = max(-ones(size(E)),E);E = min(ones(size(E)),E);theta = acos(E(1:2*n));

ResultsResults

Example GaitsExample Gaits

Name that GaitName that GaitPensive and determined.

Obligatory Gait Video 2Obligatory Gait Video 2Not in a rush.

Obligatory Gait Video 3Obligatory Gait Video 3Sad walking.

Obligatory Gait Video 4Obligatory Gait Video 4There is no stopping me.

ResultsResults“The pairwise distance between each sequence in the dataset is displayed in this plot. Each row/column of a matrix represents a sequence, and sequences corresponding to similar gaits are grouped in block rows/columns. Dark indicates a small distance, light a large distance. The minimum distance is of course along the diagonal, and for each column the next closest sequence is indicated by a circle, while the second nearest is indicated by a cross.”

ResultsResults How accurate are How accurate are

the results?the results? How does this How does this

compare with compare with human ability?human ability?

Questions?Questions?

From Discussion BoardFrom Discussion Board

Diem: How are the ARMA model and Kalman Diem: How are the ARMA model and Kalman filters related?filters related?

Mei-fang: Is there any method to find the best Mei-fang: Is there any method to find the best size training set, and the best k for classifying size training set, and the best k for classifying the new sample?the new sample?

Matt Clothier: What is the advantage of the Matt Clothier: What is the advantage of the Martin distance over the Finsler distance?Martin distance over the Finsler distance?

Neil and Mike: What do they mean by “Transient Neil and Mike: What do they mean by “Transient Actions” and why does the previous assumption Actions” and why does the previous assumption not hold for them?:not hold for them?: