Introduction to Action Recognition in Python by Bertrand Nouvel, Jonathan Kelsey, and Bernard...

Introduction to

Action Recognition

in Python@wideio

Bertrand NOUVEL, [email protected]

Jonathan KELSEY, [email protected]

Bernard HERNANDEZ, [email protected]

PYDATA LONDON 2014

mailto:[email protected]



WIDE IO - DEMOCRATISING THE BEST ALGORITHMS

INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

Outline

- (While you download the data) Forewords & Overview

- PART 1: Video-processing in python

- PART 2: The pipeline in details

- PART 3: Putting it together



FOREWORDS



GETTING READY

Python 2.x / 3.x

Numy/Scipy

PIL

OpenCV2

ANACONDA

WAKARI

GET THE ABSOLUTELY ESSENTIAL PACKAGES :

1)

2)GET THE SOURCE CODE

git clone https://bitbucket.org/wideio/pydata.git



WHO WE ARE

WIDE IO - Democratising the best algorithms

startup + consultancy



Why do Computer Vision in Python?

Computer vision is difficult.

Multiparadigm

Best community

Lots of packages

Operator

overloading

Readable

Reflexive

Lightweight

Python is 1000 slower than C++

“””



ACTION RECOGNITION



What is action recognition ?

SEMANTIC GAPKTH, Human action dataset (Laptev)

Classify actions

Classification task



Types of Systems

ACTION SPECIFICHOLISTIC

APPROACHES

FEATURE BASED

APPEARANCE

BASED

DEEP LEARNING

Many priors Less priors



The Very-Traditional Pipeline

RAW DATA FEATURE-EXTRACTION MACHINE LEARNING




VIDEO

FRAMESSPARSIFICATION

CLASSIFICATION

METHOD




VIDEO

FRAMESBAGS OF KEYPOINTS SVM

PART I

PROCESSING

VIDEOS

TOWARDS FRAMEWORKS



- PYFFMPEG (outdated)

- OPENCV

- MLT

- ON WAKARI

Reading Videos

anaconda$ vi io/player_mlt.py



- PYFFMPEG (outdated)

- OPENCV

- MLT

- ON WAKARI

Reading Videos

anaconda$ vi io/player_cv.py



Displaying ImagesRecommended Alternatives

anaconda$ python io/display_pyglet.py



Harris Corner Detector

Detecting interest points:

imsmooth=scipy.ndimage.gaussian_filter

def harris(I,alpha=0.04,si=1):

Ix,Iy = scipy.gradient(I)

H11 = imsmooth(Ix*Ix, si)

H12 = imsmooth(Ix*Iy, si)

H22 = imsmooth(Iy*Iy, si)

return ((H11*H22 - H12**2)

- alpha*(H11+H22)**2)

anaconda$ python keypoints/harris.py



Optical flow - wrapper based code

anaconda$ python keyponats/custom_feature.py



Use the two previous elements to compute keypoints that contain information about the movement

Ideas for extension: Make the same with a pyramidal approach.

Spatio-temporal keypoints

anaconda$ python keyponats/custom_feature.py

PART II

UNDERSTANDING THE KEY

ELEMENTS OF THE PIPELINE



Feature extractions



SIFT (128)

SURF (64)Descriptor vector

Feature extractions



Algorithms:

Type (density, connectivity, …)

Dimension (descriptor, position, …)

Recursive clustering

Connectivity (ward) Centroid (k-means) Density (optics)



Support Vector Machines




Weighted-Support Vector Machines

Different support




Weighted-Support Vector Machines

Different support

Different relevance (outliers)

PART III

PUTTING EVERYTHING

TOGETHER



Feature detection

Clustering

PCA Projection

BOW

Eigen Vectors & Mean

Graphic ModelModel.model

centers



Distance of projection to BOW cluster centers with metric

Mean of stack to create histogram

Normalise (Area=1)

Invert with 'discriminator' to turn into weighting

PCA project with graphic model

Feature detection stack for image



Training data

Feature computer

Feature stack

Param grid

Training Labels

SVM predict all

Testing labelsTesting data

SVM auto train

Introduction to

Action Recognition

in Python@wideio

Bertrand NOUVEL, [email protected]

Jonathan KELSEY, [email protected]

Bernard HERNANDEZ, [email protected]

PYDATA LONDON 2014




Introduction to Action Recognition in Python by Bertrand Nouvel, Jonathan Kelsey, and Bernard...

Technology

Transcript of Introduction to Action Recognition in Python by Bertrand Nouvel, Jonathan Kelsey, and Bernard...