Computer Vision GroupUniversity of California Berkeley

Recovering Human Body Configurations: Combining Segmentation and Recognition

Greg Mori, Xiaofeng Ren, Alyosha Efros and Jitendra Malik

Computer Vision GroupUniversity of California Berkeley

Problem

Input image Stick figure Support masks

Single image: No background subtraction

Computer Vision GroupUniversity of California Berkeley

Human Figures in Still Images

• Detection of humans is possible for stereotypical poses– Standing– Walking(Mohan, Papageorgiou, and Poggio PAMI 01;

Viola, Jones and Snow ICCV 03)

• But we want to do more– Wider variety of poses– Localize joint positions

Computer Vision GroupUniversity of California Berkeley

Why Is This Hard?

• Variety of poses

• Clothing

• Missing parts

• Small support for parts

• Background clutter

Computer Vision GroupUniversity of California Berkeley

Previous Work

• Exemplar approaches– Shape Contexts (Mori and Malik ECCV

2002)– Order Structure (Sullivan and Carlson

ECCV 2002)– LSH (Shakhnarovich et al. ICCV 2003)

• Part-based approaches– Rectangles (Ioffe and Forsyth, IJCV 2001)– Corners (Song et al., PAMI 2003)– Dynamic programming (Felzenszwalb and

Huttenlocher, CVPR 2000)

Computer Vision GroupUniversity of California Berkeley

Approach:Unifying Segmentation and Recognition

• Bottom-up– Detect half-limbs and torsos

• Top-down– Assemble parts into human

figure

Computer Vision GroupUniversity of California Berkeley

Why Segmentation for Recognition?

• Window-scanning (e.g. face detection)

SUPERPIXELS

SEGMENTS

• Segmentation

• How many?

Computer Vision GroupUniversity of California Berkeley

Limb/Torso Detectors• Learn limb and torso detectors

from hand-labeled data

• Cues:– Contour

• Average edge strength on boundary

– Shape• Similarity to rectangle

– Shading• x,y gradients, blurred

– Focus• Ratio of high to low frequency

energies

Computer Vision GroupUniversity of California Berkeley

Islands of Saliency

• “Partial configurations”– 3 half-limbs plus a torso

• Combinatorial search over sets of limbs and torsos T

L

326783

configurations

Computer Vision GroupUniversity of California Berkeley

Pruning Partial Configurations

• Many partial configurations are physically impossible

• Prune using global constraints (not a tree)– Proximity– Relative widths– Maximum lengths– Symmetry in clothing colour

• Results in 1000 partial configurations

Computer Vision GroupUniversity of California Berkeley

Completing Configurations

• Use superpixels to complete half-limbs– 2 or 3-limbed people

• Sort partial configurations– Use limb, torso, and

segmentation scores

• Extend missing limb(s) of best configurations

Computer Vision GroupUniversity of California Berkeley

LU ARM

RL LEGLL LEG

RL ARM

RL LEGLL LEG

Computer Vision GroupUniversity of California Berkeley

Results I

Computer Vision GroupUniversity of California Berkeley

Results II

Rank 3

Rank 3

Computer Vision GroupUniversity of California Berkeley

Addressing the Challenges

• Variety of poses– Parts-based approach, no restrictive priors

• Clothing– Shape, shading, edge cues

• Missing parts– Start with “islands of saliency”

• Small support for parts– Start with “islands of saliency”

• Background clutter– Global constraints on relative sizes of parts, colour

Computer Vision GroupUniversity of California Berkeley

All segmentations are wrong, but some segmentations are useful!