11

Building a Dictionary of Image Fragments

Zicheng Liao Ali Farhadi Yang Wang Ian Endres David Forsyth

Department of Computer Science, University of Illinois at Urbana-Champaign

2

OutlineOutline

► IntroductionIntroduction►Related WorkRelated Work►Building a Dictionary of Fragments►Applications►Conclusion

3

IntroductionIntroduction

4

IntroductionIntroduction

► Image fragment – regions that could Image fragment – regions that could represent a single object, an object in represent a single object, an object in context or a piece of a scene – form a context or a piece of a scene – form a natural representation of objects.natural representation of objects.

►Key step - to build a large, rich dictionary of image fragments automatically.

5

IntroductionIntroduction

► Four steps:Four steps: 1.G1.Generate a set of fragment proposals from

image sets. 2.Verifythe qualities of the generated fragment

proposals with a discriminative method. The selected fragments are grouped and indexed by the labels of their source images.

3.Use a clean-up procedure to remove anomalies from the dictionary within each category.

4.Matte the resulting fragments out of the training images to get the best possible boundaries.

6

Related WorkRelated Work

►Automated object segmentation (a) that it is useful to work with more than

one segmentation of a particular image, then choose good fragments

(b) that these multiple segmentations can yield estimates of support

(c) that it is possible to identify segments that seem to form a single object, without knowing what the object is

7

Related WorkRelated Work

►Exemplar-based image classification There exist methods for image

classification using region-based exemplar matching and image-based exemplar matching

8

Building a Dictionary of Fragments

►1. 1. Generating fragment proposals►2. Fragment verification►3. Dictionary clean-up►4. Matting dictionary fragments

9

Building a Dictionary of Fragments

10

Building a Dictionary of Fragments-Generating fragment

proposals►Proposing regions: Proposing regions: This step aims to

generate a large and diverse set of proposals that are likely to be object regions.

11

Building a Dictionary of Fragments-Generating fragment

proposals►Ranking proposals:

The next step of is to rank all the proposals in an image so that object regions are ranked higher than non-object regions.

Use a rank-SVM formulation based on various features computed from proposal regions, such as color, texture, geometric surfaces, boundaries, etc.

12

Building a Dictionary of Fragments-Fragment

Verification

13

Building a Dictionary of Fragments-Fragment

Verification► Flickr: These images are downloaded from Flickr.

They are mainly about humans, objects, activities, pets, and familiar scenes (indoor and outdoor).

►Caltech256: This dataset is widely used for object recognition in the computer vision literature.

►PASCAL VOC2010: This is another widely used object recognition dataset. The images are more complex than those in Caltech256.

14

Building a Dictionary of Fragments-Fragment

Verification

15

Building a Dictionary of Fragments-Fragment

Verification►A good fragment can be a whole object, a

meaningful component of a larger object , or a scene that consists of part of an scene.

►A good fragment’s effective size should cover neither too little nor too much of the entire image domain. If a fragment covers too little of the image, it may contain little discriminative information and is unlikely to be useful for image compositing. On the other hand, if a fragment covers too much of the image, its distinction from the whole image is small.

16

Building a Dictionary of Fragments-Dictionary Cleanup

►Fragments associated with the same tag (e.g. “cat”) in the source images are grouped as “cat” fragments.

►In this step we want to remove such within-class fragment anomalies from the dictionary.

17

Building a Dictionary of Fragments-Dictionary Cleanup

►How to cleanup For each new incoming fragment, we use

an adapted asymmetric region-to-image matching algorithm to measure its distance to the fragment set and count the top k (5 in experiment) best matches.

18

Building a Dictionary of Fragments-Dictionary Cleanup

19

Building a Dictionary of Fragments-Matting Dictionary

Fragments►The closed-form matting algorithm

simplifies the matting equation with a local window color smoothness assumption, and transforms the problem into a quadratic optimization problem, which can be solved efficiently via a sparse linear solver.

20

ApplicationsApplications

►1. 1. Image Classification►2. Object Localization►3. Image Composition

21

Applications- Applications- Image Classification

►Fragments may have a small advantage because contextual information creates noise, or because our fragment selection procedure will prefer images with high contrast between fragment and background.

22

Applications- Applications- Image Classification

23

Applications- Applications- Object Localization

►The spatial support of fragments can be used to accurately localize objects in a query image.

24

Applications- Applications- Object Localization

25

Applications-Image Applications-Image CompositionComposition

26

Applications-Image Applications-Image CompositionComposition

27

ConclusionsConclusions

►Use the highly localized information of fragment-based matching to do object detection.

28

►Thanks for your listening.Thanks for your listening.