Searching for Category-Consistent Features: A Computational...

Searching for Category-Consistent Features: A Computational Approach to Understanding Visual Category Representation

CHEN - PI NG YU, JUST I N MAXFI ELD, AND G REG ORY J ZEL I NSKY

CATEGORY-CONSISTENT FEATURES 1

Hierarchical Levels

Mammal • Superordinate

Dog • Basic

Golden • Subordinate Retriever


The Questions 1. What might affect the performance of categorical search? ◦ Our hypothesis: the specificity and distinctiveness of the category.

◦ Specificity and distinctiveness are quantified by categorical visual features.

2. How might the visual features of object categories be extracted? ◦ Our answer: learn a feature representation for each object category.

3. How likely is this hypothesis to be true? ◦ Collect behavioral data on categorical search performance

◦ Build the model, and learn the generative features from the data

◦ Evaluate the model’s fit against the behavioral data


Specificity & Distinctiveness Subordinate level: ◦ Very Specific

◦ Not Distinctive

Basic Level: ◦ Somewhat Specific

◦ Somewhat Distinctive

Superordinate Level: ◦ Not at all specific

◦ Very Distinctive

Subordinate Basic Superordinate

SpecificityDistinctiveness


i.e. Taxis Cars Vehicles

Search Procedure

Vehicle

Plane

Passenger Airliner

+

2500 ms 500 ms Search Display

(guidance epoch) Search Display

(verification epoch)


26 subjects, 288 trials (target present + absent)

16°

Time to Target

550

600

650

700

750

800


Time to Target

(ms)

Cue


Target Fixated First

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Subordinate BasicSuperordinate

Proportion of

Immediate Target

Fixations

Cue


Verification Time

600

700

800

900

1000

1100

1200

1300

1400


Verification Time (ms)

Cue


Model - Feature representation Learning a novel object category:

Finding the commonalities that represent the category.


i.e. What is a dragon fruit? ◦ Ellipsoid, pinkish red,

smooth texture, extruding green pedals.

A generative model: Category-Consistent-Features (CCFs).

Discriminative vs Generative



Superordinate Basic Subordinate

Vehicle Car Police Car

Taxi

Race Car

Boat Sail Boat

Cruise Ship

Speed Boat

Plane Passenger Airliner

Biplane

Fighter Jet

Truck 18 Wheeler

Fire Truck

Pickup Truck

Furniture Cabinet Kitchen Cabinet

Filing Cabinet

China Cabinet

Chair Folding Chair

Office Chair

Dining Room Chair

Bed Twin Bed

Canopy Bed

Bunk Bed

Table Coffee Table

Dining Room Table

End Table

Clothing Pants Jeans

Dress Pants

Pajama Pants

Shirt Dress Shirt

T-shirt

Long Sleeve Shirt

Hat Baseball Hat

Knit Cap

Cowboy Hat

Jacket Winter Jacket

Windbreaker

Trench Coat

Dessert Ice Cream Chocolate Ice Cream

Mint Choc. Chip Ice Cream

Strawberry Ice Cream

Pie Pecan Pie

Blueberry Pie

Lemon Meringue Pie

Category-Consistent Feature Model


Category-Consistent Features

Bag-of-Words Method

00

Step 1: Extract features and create a visual dictionary

visual words in dictionary

Step 2: Create descriptors in this common feature space for individual exemplars

bag-of-words histogram

Figure adapted from Bandara (2014)


Taxis Cars Vehicles


Histogram Visualization Taxis

Cars

Visual Words

Exe

mp

lars

1

1064

1

100

Exe

mp

lars

300

1

Visual Words 1064 1



Cars

Visual Words

Exe

mp

lars

1

1064

1

100

Exe

mp

lars

300

1

Visual Words 1064 1



Taxis Cars Vehicles

Taxis Cars Vehicles


What are the representative features (CCFs)?

High frequency, low variation


Taxis

Interq

uartile R

ange R

ule

Inverse C

oefficien

t of V

ariation


Taxis Category-Consistent Features



Dress pants

Visualized CCFs Knit caps

Sugar cookie

Sailboats


Number of CCFs = Specificity

Number of CCFs was highest at the subordinate level, approximating the specific within category similarity

What about between category distinctiveness?

0.45

0.5

0.55

0.6

0.65

55

60

65

70

75

80

85


Nu

mb

er

of

CC

Fs


Sibling Distance = Distinctiveness

boats

sailboat

cars

police car race car


Siblings: categories that share the same parent.

Specificity & Distinctiveness

Nu

mb

er

of

CC

Fs

Me

an

Sib

lin

g D

ista

nce

0.45

0.5

0.55

0.6

0.65

55

60

65

70

75

80

85



Subordinate level: ◦ Very Specific

◦ Not Distinctive

Basic Level: ◦ Somewhat Specific

◦ Somewhat Distinctive

Superordinate Level: ◦ Not at all specific

◦ Very Distinctive

Model Performance


550

600

650

700

750

800

850

900


Tim

e t

o T

arg

et

(ms)

Level in Category Hierarchy

Control

Behavioral

Model

800

900

1000

1100

1200

1300

1400


Ve

rifi

cati

on

Tim

e (

ms)

Level in Category Hierarchy

Control

Behavioral

Model

Guidance: #-of-CCFs Verification: #CCFs*Sibling-Dist

Trial-by-Trial fit


CCF model vs Subject Model (144 target present trials)

Paired t-test: correlations were not significantly different, other than the superordinate level (random first-target-fixated).

Psychological Science, in press 2016

Current work


How can we do even better? ◦ Predict categorical search performance on individual categories.

Drawbacks of the BoW-CCF model ◦ Single level of image features

◦ Hand designed features (SIFT)

Convolutional Neural Network (CNN-CCF) ◦ Hierarchical features

◦ Features learned directly from images

Convolutional Neural Networks (CNNs)


A typical neural network A Convolutional Neural Network

Searching for pulsars using image pattern recognition - Zhu, W.W. et al. Astrophys.J. 781 (2014) 2, 117 arXiv:1309.0776

http://inspirehep.net/record/1252539?ln=en

http://inspirehep.net/author/profile/Zhu, W.W.?recid=1252539&ln=en

CNN features


https://devblogs.nvidia.com/parallelforall/accelerate-machine-learning-cudnn-deep-neural-network-library/

http://cs231n.github.io/convolutional-networks/

Ventral-stream CNN-CCF


AlexNet, NIPS 2012

96 (11) 256 (51) 384 (99) 384 (131) 256 (163)

Kravitz et al. 2012

Ventral-stream CNN

442 (11) 470 (16) 213 (53) 154 (64) 71 (132)

Layer sizes are based on Felleman et al. 1991

Ventral-stream CNN-CCF


Convolutional layers FC layers

vsCNN-CCFs: the filters that are highly, and consistently activated, given images of a category.

Goal: search performance prediction for individual categories

Acknowledgments Justin Maxfield and Greg Zelinsky

Hossein Adeli & Eye Cog Lab RA’s

NSF Grants IIS-1111047 & IIS-1161876


E Y E C O G L A B

Searching for Category-Consistent Features: A Computational...

Documents

Transcript of Searching for Category-Consistent Features: A Computational...