Large dataset for object and scene recognition

A. Torralba, R. Fergus, W. T. Freeman

80 million tiny images

Ron Yanovich Guy Peled

http://royal.pingdom.com/

Internet 2012 in numbers• 7 petabytes

– How much photo content Facebook added every month.

• 300 million– Number of new photos added every day to Facebook.

• 5 billion– The total number of photos uploaded to Instagram since its start,

reached in September 2012.

• 58 – Number of photos uploaded every second to Instagram.

• 1 – Apple iPhone 4S was the most popular camera on Flickr.

• Image search is a specialized data search used to find images

• Search methods– Image meta search– Content-base image retrieval

Image Retrieval

• Search of images based on associated metadata such as keywords, text, etc.

• Google Images– The keywords for the image search are based on

the filename of the image, the link text pointing to the image, and text adjacent to the image

Image meta search

http://en.wikipedia.org/wiki/Google_Images

• The search will analyze the actual contents of the image by colors, shapes, textures etc.

• The most common method for comparing two images in content based image retrieval is using an image distance measure.

• Many CBIR systems have been developed, but the problem of retrieving images on the basis of their pixel content remains largely unsolved.

Content-based image retrieval (CBIR)

http://en.wikipedia.org/wiki/Content-based_image_retrieval

prag.diee.unica.itwww.dailydawdle.com

Why not combine both methods?

Primary goals

• 79,000,000 images collected from WWW

• Image matching similar to Google search prediction– “Did you mean?” tool

The problem• 79,000,000 images

– Large storage

– Long process time

Collecting ~80,000,000 images

• Using image search engines:– Altavista, Ask, Flickr, Cydral, Google, Picsearch and

Webshots

• 760GB on one hard disk?

www.apartmenttherapy.com

Creating image dataset

• Each image is labeled with one of the 75,062 non-abstract nouns in English,

as listed in the Wordnetlexical database.

• The result is a large semantic tree

What is WordNet

• WordNet® is a large lexical database of English Nouns, verbs, adjectives and adverbs are grouped into sets of cognitive synonyms (synsets), each expressing a distinct concept.

http://wordnet.princeton.edu

carrot

Plant root

Plant organ

Plant part

Natural object

Object, physical object

entity, physical thing

entity

mechanism

Mechanical device

sprinkler

http://www.cs.princeton.edu/courses/archive/spr07/cos226/assignments/wordnet.html

Reduce space and process time

With The size of 32X32 we can get more than 80% correct recognition rate

Reduce space and process time

• Moving from 256X256 to 32X32

Reduce space and process time

Studies on the face perception have shown that only 16X16 pixels needed for robust face recognition

This remarkable performance is also found in a scene recognition task

Reduce space and process time

• Speech recognition uses 10^6 data points.

• Current experiments in object recognition typically use 10^2 - 10^4

Reduce space and process timeHuman visual space

• ( 100 years ) * ( 30 frames per sec ) = 10^11

• All 32X32 images = 10^7400 images– Most of the images are just noise

Reduce space and process time

• We understand that 32^2 contain enough data for our purpose.

• The advantage is the ability to work with million of images (~10^8).

Statistics of low-res images

• Image matching methods:

– SSD (sum of squared differences)

– Warp

– Shift (per pixel)

Statistics of low-res images

Statistics of low-res images

Recognition

• The goal is to recognize objects and scenery by using SSD, WARP, SHIFT methods

instead of complex matching algorithms

• Given an image, the neighbors are found using some similarity measure (D-Shift)

Recognition

• Each neighbor in turn votes for its branch within the WordNet tree.

• Classification

• Image Search returns an object

Person detection

• Is it a person?

Person detection

• Standard approach :

Face detection algorithm

Not good enough

Person detection

• Better approach: Using the image DB

• More then 23% images contain pictures of people

Person detection

• Evaluating performance by two different sets of test images:

- Evaluation using randomly drawn images

- Evaluation using Altavista images

Evaluation using randomly drawn images

• Randomly drawn 1,125 images from DB• People were manually segmented on each

image

• Findings:– Large Appearance Better performance– Weaker labels Largest object

Large Appearance Better Performance

A better performance is achieved when a person’s appearance is greater than 20% of the image.

Evaluation using Altavista images

• 1,018 images drawn by searching ‘person’ label• Images classified using WordNetReordered labels

Scene recognition

• A search for images that match an entire scene rather than a specific object

• Randomly tagging 1,125 pictures to:“City” , “River” , “Field” , “Mountain”

DB Size:80,000,000800,0008,000

The larger the database, the more successful the detection rate.

Achievements

• Building a large dataset of 79 million 32x32 color labeled images.

• Showing that a simple non-parametric method, in conjunction with large dataset, can give reasonable performance on object recognition task.

• Tasks as Person detection and Scene detection perform as good as leading class specific detectors

Conclusions

It is possible to put less effort into the modeling part in object recognition (seeking to develop suitable parametric representation for recognition), while simultaneously improving the dataset itself can help to solve the same problem.

References• 80 million tiny images

– http://people.csail.mit.edu/torralba/publications/80millionImages.pdf

• ImageNet– http://wordnet.cs.princeton.edu/papers/imagenet_cvpr09.pdf

• WordNet– http://wordnet.princeton.edu/wordnet/

• Precision and recall– http://en.wikipedia.org/wiki/Precision_and_recall

• ROC curve– http://en.wikipedia.org/wiki/Receiver_operating_characteristic

• Images taken from:– http://royal.pingdom.com/– http://en.wikipedia.org/wiki/Google_Images– http://en.wikipedia.org/wiki/Content-based_image_retrieval– http:// www.prag.diee.unica.it– http:// www.dailydawdle.com– www.apartmenttherapy.com– http://www.cs.princeton.edu/courses/archive/spr07/cos226/assignments/wordnet.html

?