Generic Object and Action Detection with LARK (Locally Adaptive

Generic Object and Action Detection withLARK (Locally Adaptive Regression Kernels)

Haejong SeoUniversity of California, Santa Cruz

Mentor: Gary Bradski

Haejong Seo (summer Internship) (1)

MotivationHumanRobot InteractionRobotRobot Interaction

Where to look?

Is there any motion?

Where is a bottle of beer located?How big is this bottle?What pose is this ?

Is he/she waving hands to me?Is other PR2 approaching to me?

122

3

4

MotivationDOT (dominant orientation templates) by Stefan

1) can handle object detection2) run with a single webcam3) pretty fast

However, can not deal with


BiGGPy (binarized gradients grid pyramid) by Gary

3

1 2 4

LARK can tackle all the problems like

Motivation & Goal

static saliency object detection

spacetime saliency action detection


Develop fast and robust detection systems in open sourse

1

2 4

3

OutlineOutline

LARK Overview

Saliency Detection

Object Detection

Conclusion

Spacetime Saliency Detection

Action Detection

1

2

3

4

5

6

LARK (locally adaptive regression kernels)


●Euclidean distance vs. Geodesic distance

Image as a Surface Embedded in the Euclidean 3‐space

Arclength on the surface

Chain rule


LARK selfsimilarity →


LARK (example)


LARK (speedup)Step1: downsample by a factor of 4

Step 2: interpolate C = [C11, C12, C22] after computing in a lower scale

C11 C22 C12


0.02 sec (70 times faster)

Saliency Detection


LARK selfresemblance

Saliency map

thresholding

Saliency Detection (video)


Object Detection


Compute LARKCompute LARK

templatetemplate

imageimage

Object Detection (speedup)

Use saliency to reduce search space


Face Detection (video)


One template Three templates

Object Detection (video)


Door knob PR2

Drawing Small robot

Three templates

3D Object Detection (speedup)

Pyramid searchPyramid searchTree structure for template


3D Object Detection (video)


CD case mouse

naked organizer

3D Object Detection (video)


Two objects Three objects

3D LARK selfsimilarity in 3D→


Spacetime Saliency Detection


Spacetime Saliency (video)


Action Detection


template

Input video

LARKs(30~35 frames)

Action Detection (speedup)


spacetime saliencypyramid search

5 frames 5 frames

35 frames

7 frames of 3D LARK (3x3 (space)x5 (time))

Action Detection (video)


4 actions

sitting down

moving closer

boxing

waving

Code Availability

Package larks: service that trains object templates and detects objects locations and poses (available now)

→ stacks/object_recognition_experimental/larks


Package saliency: service that provides salient regions in images and videos (will be available)

→ cturtle/wgrospkgunreleased/sandbox/saliency

Package actiondetection: service that detect generic human actions in videos (will be available)

→ cturtle/wgrospkgunreleased/sandbox/actiondetection

Discussion & Future Work

Use partsbased detection to deal with occlusion (Steve Gould)

Use a tracking algorithm to avoid blinking effects

Improve scalability build a common tree for all the objects→


Learn threshold values for each object and action

Use LARK as a post filter for BiGGPy

Thank you!

Any Questions?

Generic Object and Action Detection with LARK (Locally Adaptive

Documents

Transcript of Generic Object and Action Detection with LARK (Locally Adaptive