[1] P. Arbelaez, M. Maire, C. Fowlkes, and J. Ma-lik. Contour Detection and Hierarchical ImageSegmentation. PAMI, 2011.

[2] M. Maire, S. X. Yu, and P. Perona. Object De-tection and Segmentation from Joint Embeddingof Parts and Pixels. ICCV, 2011.

[3] D. Martin, C. Fowlkes, D. Tal, and J. Malik. ADatabase of Human Segmented Natural Imagesand its Application to Evaluating SegmentationAlgorithms and Measuring Ecological Statistics.ICCV, 2001.

[4] M. Spain and P. Perona. Measuring and Predict-ing Object Importance. IJCV, 2010.

ONR MURI N00014-10-1-0933 and ARO/JPL-NASA Stennis NAS7.03001 supported this work. Part of Stella Yu’s work was supported by NSF CAREER IIS-1257700.

Benchmarks

Image Groundtruth UCM gPb-UCM [1] Residual

• Construct groundtruth Ultrametric Contour Map (UCM):– Weight each boundary by level at which it appears in object-part hierarchy– Determine boundary visibility by region tree traversal• Does machine hierarchical segmentation [1] respect object-part containment?

– Residuals (above) show differences by:∗ Type: false positive / false negative / incorrect level∗ Severity: color intensity reflects magnitude of error

– Plots (below) measure recovery order of groundtruth boundaries:∗ Ideally recover top-level objects, then parts, then subparts (levels 1, 2, 3)∗ gPb-UCM only recovers groundtruth hierarchy on simpler scenes (portraits)

All

Sce

nes

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Overall Boundary Recall

Leve

l Rec

over

y F

ract

ion

Boundary Recovery Order by Hierarchy Level

Level 1Level 2Level 3

Port

rait

Sce

nes

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Overall Boundary Recall

Leve

l Rec

over

y F

ract

ion

Boundary Recovery Order by Hierarchy Level

Level 1Level 2Level 3

Dataset

Par

tsO

bjec

tsIm

age

• Complete labeling of 97 complex scenes photographed by artist Stephen Shore [4]• Compared to Berkeley Segmentation Dataset [3]: more objects, greater scale range

Interactive Labeling

A

BDC

• Browse object-part hierarchy by expanding/collapsing visible subtrees (A)• Drag and drop to rearrange region hierarchy (B)• Fade occlusion layers by depth to visualize figure/ground ordering (C)• Interactively edit regions (D)

– Editor enforces parent-child region containment invariant– Superpixel selection brush speeds region definition:

Image Superpixels Click Drag Release Touch-up

Scene ModelObject-Part ⇒

⇐OcclusionOrdering

man

head

shirt

arm

watch

hand

arm

envelope

stamp

label glove

Objects/Parts

Figure/G

round

Gnd

Fig

glove

envelope

stamp

label

man

head

V

shirt

arm

V

watch

hand

arm glove

• Image⇒ regions {R1, R2, ..., Rn} (Ri, Rj possibly overlapping)• Map each region Ri to a node in doubly-ordered tree:

– Parent-child node relationships correspond to object-part containment– Relative ordering of sibling nodes resolves occlusion ambiguities• Tree traversal:

– Recovers object-part hierarchy– Converts local occlusion relationships into global figure/ground ordering• Virtual nodes (dotted ovals): parts without visible boundaries• Virtual links (dotted arrows): remap occlusion ordering (handle self-occlusion)

Overview• Annotate multiple modalities:

– Objects, parts, subparts– Object-part containment– Segmentation– Occlusion (figure/ground)

• Unifying abstraction: region trees• Web-based annotation tool:

– Computer-assisted segmentation• Object segmentation dataset & benchmark• Motivation: joint detection & segmentation [2]

appleCCCCCCCCW

glassesPPPPPPPPq

hat�

face�

torso��

�����

eyes HHHHHj

PPPPPPPPPq

beardPPPPPPq

finger

?

shirt��

����

tiePPPP

PPi

jacket����������������

Fig

Gnd

Image Objects/Parts Subparts Figure/Ground Ultrametric Contour Map

Michael Maire1, Stella X. Yu2, and Pietro Perona11California Institute of Technology - Pasadena, CA 91125 2University of California at Berkeley / ICSI - Berkeley, CA 94704

Hierarchical Scene Annotation