Part 4: Combined segmentation and recognitionby Rob Fergus (MIT)

AimGiven an image and object category, to segment the objectSegmentation should (ideally) be shaped like the object e.g. cow-like obtained efficiently in an unsupervised manner able to handle self-occlusionSegmentationObjectCategory ModelCow ImageSegmented CowSlide from Kumar 05

Feature-detector view

Examples of bottom-up segmentation Using Normalized Cuts, Shi & Malik, 1997Borenstein and Ullman, ECCV 2002

Jigsaw approach: Borenstein and Ullman, 2002

Perceptual and Sensory Augmented ComputingInterleaved Object Categorization and Segmentation

Implicit Shape Model - Liebe and Schiele, 2003Liebe and Schiele, 2003, 2005

Random Fields for segmentationI = Image pixels (observed)h = foreground/background labels (hidden) one label per pixel = ParametersPriorLikelihoodPosteriorJointGenerative approach models joint Markov random field (MRF)

2. Discriminative approach models posterior directly Conditional random field (CRF)

Generative Markov Random Field I (pixels)Image PlaneijPrior has no dependency on I

Conditional Random FieldLafferty, McCallum and Pereira 2001PairwiseUnary Dependency on I allows introduction of pairwise terms that make use of image.

For example, neighboring labels should be similar only if pixel colors are similar Contrast termDiscriminative approache.g Kumar and Hebert 2003

OBJCUT (shape parameter)Kumar, Torr & Zisserman 2005PairwiseUnary is a shape prior on the labels from a Layered Pictorial Structure (LPS) model

Segmentation by:

- Match LPS model to image (get number of samples, each with a different pose

Marginalize over the samples using a single graph cut [Boykov & Jolly, 2001]Label smoothnessContrastDistance from Color Likelihood

OBJCUT:Shape prior - - Layered Pictorial Structures (LPS)Generative modelComposition of parts + spatial layout

Layer 2Layer 1Parts in Layer 2 can occlude parts in Layer 1Spatial Layout(Pairwise Configuration)Kumar, et al. 2004, 2005

OBJCUT: ResultsIn the absence of a clear boundary between object and backgroundSegmentationImageUsing LPS Model for Cow

Levin & Weiss [ECCV 2006] Segmentation alignment with image edgesConsistency with fragments segmentation

Winn and Shotton 2006Layout Consistent Random Field

Layout consistencyNeighboring pixels(p,q)?(p,q+1)(p,q)(p+1,q+1)(p-1,q+1)Layout consistentWinn and Shotton 2006

Layout Consistent Random FieldWinn and Shotton 2006

Stability of part labellingPart color key

Object-Specific Figure-Ground SegregationStella X. Yu and Jianbo Shi, 2002

Image parsing: Tu, Zhu and Yuille 2003

Image parsing: Tu, Zhu and Yuille 2003

Segment out all the cars.fused tree model for carsUnseen imageTraining imagesSegmented CarsSegmentation TreesOverviewMultiscale Seg.Todorovic and Ahuja, CVPR 2006Slide from T. Wu

LOCUS modelDeformation field DPosition & size T Class shape Class edge sprite o,oEdge image eImageObject appearance 1Background appearance 0Mask mShared between imagesDifferent for each imageKannan, Jojic and Frey 2004Winn and Jojic, 2005

In this section: brief paper reviewsJigsaw approach: Borenstein & Ullman, 2001, 2002Concurrent recognition and segmentation: Yu and Shi, 2002Image parsing: Tu, Zhu & Yuille 2003 Interleaved segmentation: Liebe & Schiele, 2004, 2005OBJCUT: Kumar, Torr, Zisserman 2005LOCUS: Winn and Jojic, 2005LayoutCRF: Winn and Shotton, 2006Levin and Weiss, 2006Todorovic and Ahuja, 2006

SummaryStrengthExplains every pixel of the imageUseful for image editing, layering, etc.

IssuesInvariance issues(especially) scale, view-point variationsInference difficulties

Conditional Random Fields for SegmentationSegmentation map xImage ILow-level pairwise termHigh-level local termPixel-wise similarity

Object-Specific Figure-Ground SegregationSome segmentation/detection resultsYu and Shi, 2002

Multiscale Conditional Random Fields for Image LabelingXuming He Richard S. Zemel Miguel A . Carreira-PerpinanConditional Random Fields for ObjectRecognitionAriadna Quattoni Michael Collins Trevor Darrell

OBJCUTProbability of labelling in addition has Unary potential which depend on distance from (shape parameter)D (pixels)m (labels) (shape parameter)Image PlaneObject CategorySpecific MRFxymxmyUnary Potentialx(mx|)Kumar, et al. 2004, 2005

Localization using features

Levin and Weiss 2006Levin and Weiss, ECCV 2006

Results: horses

Results: horses

Cows: ResultsSegmentations from interest points

Single-frame recognition - No temporal continuity used!Liebe and Schiele, 2003, 2005

Examples of low-level image segmentationNormalized Cuts, Shi & Malik, 1997Borenstein & Ullman, ECCV 2002

Jigsaw approachEach patch has foreground/background mask

LayoutCRF

Segmentation

Interpretation of p(figure) mapper-pixel confidence in object hypothesisUse for hypothesis verificationLiebe and Schiele, 2003, 2005

Different occlusions preserves ordering, deformations preserve ordering

*Different occlusions preserves ordering, deformations preserve ordering

*Edge weight larger at image edges**Write down the contribution part of this paperEmphasise class model (shared) all other variables per-image. Emphasise LEARN EVERYTHING SIMULTANEOUSLY.