Face Tracking and Person Action Recognition - Update

M4 meeting@Prague 22-23.01.2004

Institute for Human-Machine CommunicationMunich University of Technology

Sascha Schreiber

Face Tracking and Person Face Tracking and Person Action Recognition - UpdateAction Recognition - Update

M4 [email protected]

Sascha Schreiber2/14


• Recapitulation of methodology for action recognition

• Face tracking with I-Condensation

• Recognition performance comparison on actions from the m4 dataset

• Kalman filtering of occluded gestures

• Outlook

Overview




Person Action Recognition

Extraction of person locations

Temporal segmentation

Feature calculation

Classification of segments

Face detection/Blob tracking

Global Motion Features

Bayesian Information Criterion

Hidden Markov Models

Actions, timestamps





Extraction of person Extraction of person locationslocations


Feature calculation

Classification of segments

Face detection/Blob trackingFace detection/Blob tracking



Hidden Markov Models

Actions, timestamps




Face Tracking

Particle filtering with ICondensation( ) ( ){( , ), 1,..., }i it tx i N ( ) ( ){( , ), 1,..., }i it tx i N • N weighted particles

( ) ( ) ( )1 ( | )i i i

t t t tp y x ( ) ( ) ( )1 ( | )i i i

t t t tp y x • Updating using their likelihood

• Sampling from prediction density

- Standard Condensation sampling

- Sampling from importance function for reinitialisation

- Importance sampling with weighting correction factor

Introduction of importance function: skin color distribution

Automatic initialization by pyramid sampling and MLP classification




Performance of Face Tracking

Standard Condensation ICondensation

Demonstration of difference between:





Extraction of person locations


Feature calculation

Classification of segmentsClassification of segments

Face detection/Blob tracking



Hidden Markov ModelsHidden Markov Models

Actions, timestamps




IDIAP training data (TRN 01-30), IDIAP test data (TST 01-30)

Continuous HMMs (6 states, 3 mixtures)

Sit down Stand up NoddingShaking head

Writing Pointing Score

Sit down 9 0 0 0 0 1 90%

Stand up 1 9 0 0 1 3 64%

Nodding 2 1 215 58 6 8 74%

Shaking head

0 0 22 16 4 1 37%

Writing 0 0 38 20 468 24 85%

Pointing 0 0 2 0 0 70 97%

Overall 80%

Recognition Performance m4




IDIAP training data (TRN 01-30), IDIAP test data (TST 01-30)

Discrete HMMs (6 states, codebook 1500)

Sit down Stand up NoddingShaking head

Writing Pointing Score

Sit down 9 0 0 0 0 1 90%

Stand up 3 10 0 0 0 1 71%

Nodding 1 3 251 2 33 0 87%

Shaking head

0 0 30 2 11 0 5%

Writing 0 0 28 0 515 7 94%

Pointing 4 1 4 0 6 57 79%

Overall 86%

Recognition Performance m4




Occluded Gestures

Classification result

Classification

Compensationof occlusion

Stream-segmentation

Feature-extraction

Smoothed featurestream

Featurestream

Segmented Featurestream

Video-stream

Occlusion

Scenario: Person walking on front of a tracked object




Occluded Gestures

Application for Kalman filtering:

• Calculation of an estimate

1ˆ ˆk kx Ax

Time update equation

ˆ ˆ ˆ( ) k k k k kx x K z HxMeasurement update equation

• Discrete-time process: 1 1 k k kx Ax w

k k kz Hx v

ˆkx




Occluded Gestures

xkˆ)1(

yk Kalman-

filterKalman-

filterKalman-

filter

xkn

ˆ)(

xkˆ)2(

• N action-specialized Kalman-Filters, each trained for a special gesture to be recognized by the HMM

Improving featurestream by smoothing with :

xkˆ

yk Kalman-

filter

• One general Kalman-Filter for the disturbed featurestream




Performance of Kalman filtering

Score

Featurestream unoccluded & unfiltered 79,86%

Featurestream occluded & unfiltered 56,75%

Featurestream occluded & filtered (general) 57,98%

Featurestream occluded & filtered (specialized) 60,12%

IDIAP training data (TRN 01-30), IDAP test data (TST 01-30)

Continuous HMMs (6 states, 3 mixtures)




• Implementation of extended Kalman filter• Head orientation tracking• Integration of face recognition into particle filter• Further improvement of action detection on m4 data• Connection to Meeting Segmentation / Multimodal

Recognizer

Outlook

M4 meeting@Prague 22-23.01.2004


Face Tracking and Person Face Tracking and Person Action Recognition - UpdateAction Recognition - Update

Sascha Schreiber

Face Tracking and Person Action Recognition - Update

Documents

Transcript of Face Tracking and Person Action Recognition - Update