Large-scale CelebA face attribute dataset

• 200K celebrity images, each with 40 attributes

• Highly imbalanced: average positive class rate 23%

• Total accuracy =𝑡𝑝 + 𝑡𝑛

𝑁𝑝 + 𝑁𝑛Balanced accuracy =

1

2

𝑡𝑝

𝑁𝑝+

𝑡𝑛

𝑁𝑛

Edge detection on BSDS500 dataset

• Retrieve from 2M edge label patches with long-tail distribution

Learning Deep Representation for Imbalanced ClassificationChen Huang1,2, Yining Li1, Chen Change Loy1, Xiaoou Tang1

1The Chinese University of Hong Kong 2SenseTime Group Limited

{chuang, ly015, ccloy, xtang}@ie.cuhk.edu.hk

Data imbalance is common in visual classification

1. Motivation

Wearing

hat

Not wearing

hat

…

Minority class

Majority class

Face attribute example

Deep embedding: Class-level cluster- & class-level constraint

Study traditional re-sampling and cost-sensitive learning scheme

2. Main Idea

Triplet embedding

Class 1

minority

Class 2

majority

Class 2 majority

Quintuplet embedding

Class 1 minority

…

Cluster j

Cluster 2

Cluster 1 Cluster 1

𝐷 𝑓 𝑥𝑖 , 𝑓 𝑥𝑖𝑝+

< 𝐷(𝑓 𝑥𝑖 , 𝑓(𝑥𝑖𝑝−))

< 𝐷(𝑓 𝑥𝑖 , 𝑓(𝑥𝑖𝑝−−

)) < 𝐷(𝑓 𝑥𝑖 , 𝑓(𝑥𝑖𝑛))

𝑥𝑖 – an anchor

𝑥𝑖𝑝+

– the anchor’s most distant within-cluster neighbor

𝑥𝑖𝑝−

– the nearest within-class neighbor of the anchor, but from a different cluster

𝑥𝑖𝑝−−

– the most distant within-class neighbor of the anchor

𝑥𝑖𝑛 – the nearest between-class neighbor of the anchor

Triple-header hinge loss

Network architecture

• Equal class re-sampling & class costs assignment in batches

Training step

3. Large Margin Local Embedding (LMLE)

s.t.:

● Clustering by k-means

● Generate quintuplets from

cluster & class membership

● Re-sample batches equally from each class

● Forward their quintuplets to CNN to

compute loss

● Back-propagation

Feature-based clustering

Feature learning/updating

Every 5000 iterations

CNN

CNN

CNN

CNN

CNN

Trip

le-h

ead

er h

ing

e lo

ss

Mini-

batches

Training

samples …

EmbeddingQuintuplet

Shared parameters

Large margin cluster-wise kNN: fast & imbalance resistant

4. Cluster-wise kNN search

-8 -6 -4 -2 0 2 4 6 8-5

-4

-3

-2

-1

0

1

2

3

4

5

-15 -10 -5 0 5 10-20

-15

-10

-5

0

5

10

15

-10 -8 -6 -4 -2 0 2 4 6 8-8

-6

-4

-2

0

2

4

6

8

NC2

NC3

NC4

NC5

PC1

PC2

NC1

-8 -6 -4 -2 0 2 4 6 8-5

-4

-3

-2

-1

0

1

2

3

4

5

-15 -10 -5 0 5 10-20

-15

-10

-5

0

5

10

15

-10 -8 -6 -4 -2 0 2 4 6 8-8

-6

-4

-2

0

2

4

6

8

NC2

NC3

NC4

NC5

PC1

PC2

NC1

-8 -6 -4 -2 0 2 4 6 8-5

-4

-3

-2

-1

0

1

2

3

4

5

-15 -10 -5 0 5 10-20

-15

-10

-5

0

5

10

15

-10 -8 -6 -4 -2 0 2 4 6 8-8

-6

-4

-2

0

2

4

6

8

NC2

NC3

NC4

NC5

PC1

PC2

NC1

-8 -6 -4 -2 0 2 4 6 8-5

-4

-3

-2

-1

0

1

2

3

4

5

-15 -10 -5 0 5 10-20

-15

-10

-5

0

5

10

15

-10 -8 -6 -4 -2 0 2 4 6 8-8

-6

-4

-2

0

2

4

6

8

NC2

NC3

NC4

NC5

PC1

PC2

NC1

Class 1: cluster 1

Class 1: cluster 2

Class 2: cluster 1

Class 2: cluster 2

Class 2: cluster 3

Class 2: cluster 4

Class 2: cluster 5

-8 -6 -4 -2 0 2 4 6 8-5

-4

-3

-2

-1

0

1

2

3

4

5

-15 -10 -5 0 5 10-20

-15

-10

-5

0

5

10

15

-10 -8 -6 -4 -2 0 2 4 6 8-8

-6

-4

-2

0

2

4

6

8

NC2

NC3

NC4

NC5

PC1

PC2

NC1

-8 -6 -4 -2 0 2 4 6 8-5

-4

-3

-2

-1

0

1

2

3

4

5

-15 -10 -5 0 5 10-20

-15

-10

-5

0

5

10

15

-10 -8 -6 -4 -2 0 2 4 6 8-8

-6

-4

-2

0

2

4

6

8

NC2

NC3

NC4

NC5

PC1

PC2

NC1

DeepID2 Triplet Our LMLE

5. Results

Total acc. Balanced acc.

Triplet-kNN 83 72

Anet 87 80

LMLE-kNN 90 84

Ground truthSketchToken

ODS 0.73

DeepContour

ODS 0.76

LMLE-kNN

ODS 0.78

6. Conclusion

Cluster- & class-level quintuplets preserve both locality

across clusters and discrimination between classes,

irrespective of class imbalance

Large margin classification by fast cluster-wise kNN search