Binary code-based human detection

Yuji Yamauchi, Hironobu Fujiyoshi Chubu university

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Human detection

•Classify and locate human in an image

2

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Basic approach

3

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Basic approach

4

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Basic approach

Classify cropped image

5

識別器

classifier

Pos

Pos

Neg

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Basic approach

識別器

Pos

Pos

Neg

Classify cropped image

6

Detection result

classifier

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Basic approach

識別器

Classify cropped imageDetector

Detection result

7

classifier

Pos

Pos

Neg

Clustering result

識別結果の統合処理

Clustering

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Processes of training and testing

Training samplesHuman Background

8

Training

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Processes of training and testing

Training samplesHuman Background

Compute features

9

Training

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Processes of training and testing

Training samples

Compute features

Human Background

Training classifier

Classifier

10

Training

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Processes of training and testing

Training samples

Compute features

Classifier

Training classifier

Unknown sample

Human Background

Compute features

11

TrainingTesting

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Processes of training and testing

Training samples

Compute features

Classifier

Training classifier

Unknown sample

Human Background

Compute features Human / non-human

12

TrainingTesting

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Why human detection is difficult?

Appearance Pose

Viewpoint Occlusion Background clutter

Due to combined multiple factors, variance of appearances of human images become large

13

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Image local features

Image local features based on gradients in local region 14

(d) HOG feature [Dalal CVPR2005]Input image Block Gradients HOG

(a) EOH feature [Levi CVPR2004] (b) Edgelet feature [Wu ICCV2005]

Line Arc Symmetric pair

Input image Edges

Templates

(c) LBP histogram feature [Mu CVPR2008]

Input image

Gradients

vs.

patch

LBP

50 105

95255 200

80220180

75 0 1

1 1

01 1

0

100 80

110100 150

10080 90

220 0 0

0 1

00 0

1

00000000

11111111

Input image 3x3 pixels LBP Histogram

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Memory usage for HOG feature

•HOG feature • Gradient magnitude accumulates in each orientation within a local region

15

# of features : 3,360 dim. Floating type : 8 byte

Image size : 640 x 480 pixels # of windows : 50,000

26.8KB 1.25GB

Memory usage per a window Memory usage per a image

Imput image Gradients image gradient in each pixel Histogram of oriented gradientsCell Orientation

Magnitude

1 2 3 4 5 6 7 8 9

Block

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features 16

To achieve implementation to low end device, the memory usage have to be reduce

For example - Cyclone Ⅲ - Logic cells : 119,088 - Memory : 0.48MB

•For implement to a low end device

Practical application of human detection

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Reduction method of memory usage

•Reduction of feature dimension • Vector Quantization • Feature dimensions are reduced by clustering

• Principal component analysis • By projection to low-dimensional subspace, feature dimensions are reduced

!!!

•Quantization to low bit • Scalar quantization • Represent a feature at few bits

• Binary code

17

Minimal loss of information Computational cost is high

Computational cost is low Severe loss of information

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Binarized HOG feature (B-HOG)

•B-HOG feature is obtained by thresholding •Represent by unsigned integer type

18

11100001

Input image

B-HOG feature

HOG feature

ThresholdC1

B-HOG feature can reduce memory usage to 1/8 Threshold is necessary to be optimized

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Relational HOG feature (R-HOG)

•Comparing two HOGs •Normalization is unnecessary in HOG

19

Threshold is unnecessary R-HOG feature can reduce computational cost If obtained similar histograms, binary become unstable

Input image

00111110R-HOG feature

HOG feature

>C1

C2

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Difference

0

1

Magnitude

Magnitude

Orientation

Orientation1 2 3 4 5 6 7 8

1 2 3 4 5 6 7 8

0 方向シフト 10011111

Problem of R-HOG feature

•If obtained similar histograms, binary become unstable

20

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Shifted R-HOG feature (SR-HOG)

•Orientation in a histogram is shifted

21

Difference

0

1

Magnitude

Magnitude

Orientation

Orientation8 1 2 3 4 5 6 7

1 2 3 4 5 6 7 8

0 orientation shift 100111111 orientation shift 11000011

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Shifted R-HOG feature (SR-HOG)

•Orientation in a histogram is shifted

22

Difference

0

1

Magnitude

Magnitude

Orientation

Orientation2 3 4 5 6 7 8 1

1 2 3 4 5 6 7 8

0 orientation shift 100111111 orientation shift 11000011

7 orientation shift 01111000

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Effect of SR-HOG feature

23

Difference

0

1

Magnitude

Magnitude

Orientation

Orientation1 2 3 4 5 6 7 8

1 2 3 4 5 6 7 8

0 orientation shift : 10011111

Difference

0

1

Orientation4 5 6 7 8 1 2 3

Magnitude

5 orientation shift : 111110000

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Experimental overview

•Comparison methods • HOG feature + Real AdaBoost • B-HOG feature + Real AdaBoost • R-HOG feature + Real AdaBoost • SR-HOG feature + Real AdaBoost !

•Dataset • INRIA Person Dataset • http://pascal.inrialpes.fr/data/human/ !

•Evaluation method • Detection Error Tradeoff (DET) carve

24

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

INRIA Person Dataset

•Training samples • Positive sample : 2416 • Negative sample : 12180

25

•Testing samples • Positive sample : 1126 • Negative sample : about 2 million

Positive sample

Negative sample

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Experimental results

26

HOG B-HOGR-HOGSR-HOG

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Memory usage and computational cost

•Comparing necessary memory usage in feature extraction •Comparing the processing time until classification by feature extraction

27

Memory and computational cost per 1 window (64 x 128 pixels)

•R-HOG and S-HOG features can reduce computational cost to about 50% •Binary code can reduce memory usage to about 1/8

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Problems of binary code

•Quantization residual (QR) • A lot of information drops out !!!!

•Binary code is represented at discrete variables

28

閾値

HOG特徴量

量子化残差

00111100B-HOG特徴量

00111100

01011000

閾値

閾値

HOG特徴量

HOG特徴量

B-HOG feature

Quantization residual

B-HOG feature

B-HOG feature

HOG

HOG

HOG

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Proposed method

•Binary code • Binarizing a feature represented by real value

A lot of memory usage can be reduced Information drops out Binary code is represented at discrete variables

29

•Classifier introducing transition likelihood model based on quantization residual(QR) • A transition likelihood model represents the likelihood of an observed binary code x transitioning to another binary code x ’ Effective utilization of quantization residual Relationship between binary codes is represented Computational cost and memory usage does not increase

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Classifier Introducing Transition Likelihood Model Based on QR

•Relationship between binary codes is represented by transition likelihood model

30

(a) 遷移尤度モデル Transition likelihood model

HOG特徴量HOG

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Classifier Introducing Transition Likelihood Model Based on QR

•Relationship between binary codes is represented by transition likelihood model

31

(a) 遷移尤度モデル

HOG特徴量

Transition likelihood model

HOG

1111000100111110111000012値符号列 x

量子化残差

Binary code x

Quantization residual

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Classifier Introducing Transition Likelihood Model Based on QR

•Relationship between binary codes is represented by transition likelihood model

32

(a) 遷移尤度モデル

HOG特徴量

1111000100111110111000012値符号列 x

量子化残差 Transition likelihood model

Binary code x

HOG

Quantization residual

高い

遷移尤度

000000000000000100000010

11111111

000000000000000100000010

11111111

Transition likelihood

High

Low

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Classifier Introducing Transition Likelihood Model Based on QR

•Relationship between binary codes is represented by transition likelihood model

33

(a) 遷移尤度モデル

HOG特徴量

1111000100111110111000012値符号列 x

量子化残差

高い

遷移尤度

000000000000000100000010

11111111

000000000000000100000010

11111111 Transition likelihood model

Binary code x

HOG

Quantization residual

Transition likelihood

High

Low

2値符号列 x11000001Binary code x

(b) 識別器(b) 識別器 Classifier

入力画像Input image

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Classifier Introducing Transition Likelihood Model Based on QR

•Relationship between binary codes is represented by transition likelihood model

34

(a) 遷移尤度モデル

(b) 識別器

HOG特徴量

1111000100111110111000012値符号列 x

量子化残差

高い

遷移尤度

000000000000000100000010

11111111

000000000000000100000010

11111111

入力画像

2値符号列 x11000001

遷移尤度

× 0.06

× 0.02

× 0.12× 0.62

遷移尤度

Binary code x

HOG

Quantization residual

Transition likelihood

High

Low

Binary code x

Input image

00000000

11111111

0000000111000001

遷移後の 2値符号列 x’

-0.17 × 0.06

-0.68 × 0.02

0.02 × 0.120.43 × 0.62

対数オッズ　Binary code x’ Log odds Transition likelihood

+ 1.6弱識別器 h(x)Weak classifier

Transition likelihood model

Classifier

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Transition likelihood model

•A transition likelihood model represents the likelihood of an observed binary code x transitioning to another binary code x ’

35

111000012 値符号列 x

遷移後の 2値符号列 x’

00000000

11111111

00000001

111000010.45,0.50,0.40,0.20,0.40,0.50,0.25,0.90

量子化残差Quantization residuals

binary code x

Binary code x’ after transition prediction

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Transition likelihood model

•A transition likelihood model represents the likelihood of an observed binary code x transitioning to another binary code x ’ Step1. Compute non-invert scores z of binary

36

111000012 値符号列 x

遷移後の 2値符号列 x’

00000000

11111111

00000001

111000010.45,0.50,0.40,0.20,0.40,0.50,0.25,0.90

量子化残差 ２値符号の非反転度

0.6,0.4,0.6,1.2,1.4,1.6,1.2,0.20.6,0.4,0.6,1.2,1.4,1.6,1.2,1.8

1.4,1.6,1.4,1.2,1.4,1.6,1.2,1.8

1.4,1.6,1.4,0.8,0.6,0.4,0.8,1.8Quantization residuals

binary code x

Binary code x’ after transition prediction

Non-invert scores

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Non-invert score z of binary

•Probability of transition from x to x’ Point1. Whether or not the binary invert Point2. magnitude of the quantization residual q

37

0.0

2.0

-1.0 0.0 1.0

Quantization Residual q

non-invert score z

1.0

Concave function F()Convex function F()

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Transition likelihood model

•A transition likelihood model represents the likelihood of an observed binary code x transitioning to another binary code x ’ Step1. Compute non-invert scores z of binary Step2. Compute transition scores e of binary code

38

111000012 値符号列 x

遷移後の 2値符号列 x’

00000000

11111111

00000001

111000010.45,0.50,0.40,0.20,0.40,0.50,0.25,0.90

量子化残差 ２値符号の非反転度

0.6,0.4,0.6,1.2,1.4,1.6,1.2,0.20.6,0.4,0.6,1.2,1.4,1.6,1.2,1.8

1.4,1.6,1.4,1.2,1.4,1.6,1.2,1.8

1.4,1.6,1.4,0.8,0.6,0.4,0.8,1.8 0.88遷移スコア

18.21

0.840.09

Transition scores

Quantization residuals

binary code x

Binary code x’ after transition prediction

Non-invert scores

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Transition likelihood model

•A transition likelihood model represents the likelihood of an observed binary code x transitioning to another binary code x ’ Step1. Compute non-invert scores z of binary Step2. Compute transition scores e of binary code Step3. Create transition likelihood distribution E

39

111000012 値符号列 x

遷移後の 2値符号列 x’

00000000

11111111

00000001

111000010.45,0.50,0.40,0.20,0.40,0.50,0.25,0.90

量子化残差 ２値符号の非反転度

0.6,0.4,0.6,1.2,1.4,1.6,1.2,0.20.6,0.4,0.6,1.2,1.4,1.6,1.2,1.8

1.4,1.6,1.4,1.2,1.4,1.6,1.2,1.8

1.4,1.6,1.4,0.8,0.6,0.4,0.8,1.8 0.88遷移スコア

18.21

0.840.09

Quantization residuals

binary code x

Binary code x’ after transition prediction

Non-invert scores Transition scores

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Create transition likelihood distribution E

•Create transition likelihood distribution E from transition scores e of all training samples I

40

00000000

10000000

11111111

00000000

10000000

11111111

input binary codes xB

ina

ry c

od

es a

fter tra

nsitio

n p

red

ictio

n

x’

0111111101111111

Low

High

Tra

nsitio

n lik

elih

oo

d

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Classifier Introducing Transition Likelihood Model Based on QR

•Relationship between binary codes is represented by transition likelihood model

41

(a) 遷移尤度モデル

(b) 識別器

HOG特徴量

1111000100111110111000012値符号列 x

量子化残差

高い

遷移尤度

000000000000000100000010

11111111

000000000000000100000010

11111111

入力画像

2値符号列 x11000001

遷移尤度

× 0.06

× 0.02

× 0.12× 0.62

遷移尤度

Binary code x

HOG

Quantization residual

Transition likelihood

High

Low

Binary code x

Input image

00000000

11111111

0000000111000001

遷移後の 2値符号列 x’

-0.17 × 0.06

-0.68 × 0.02

0.02 × 0.120.43 × 0.62

対数オッズ　Binary code x’ Log odds Transition likelihood

+ 1.6弱識別器 h(x)Weak classifier

Transition likelihood model

Classifier

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Classifier introducing transition likelihood model

•Week classifier h(x) in Real AdaBoost

42

Transition likelihood mode P(x’| x)

W : Probability density function of each class

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Classifier introducing transition likelihood model

•Week classifier h(x) in Real AdaBoost

43

Transition likelihood mode P(x’| x)

➡Weak classifier h(x) predict transition of binary code x➡By using lookup table, computational cost is same asconventional method

W : Probability density function of each class

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Classifier introducing transition likelihood model

•Week classifier h(x) in Real AdaBoost

44

Transition likelihood mode P(x’| x)

➡Weak classifier h(x) predict transition of binary code x

•P(x’| x) is unobservable, therefore E (x’| x) is commuted

➡By using lookup table, computational cost is same asconventional method

W : Probability density function of each class

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Experimental results

45

HOGB-HOGR-HOG

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

HOGB-HOGR-HOGB-HOG based proposed method

Experimental results

46

Proposed method enables pedestrian detection that is more accurate than that of previous methods

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Experimental results

47

Proposed method enables pedestrian detection that is more accurate than that of previous methods

HOGB-HOGR-HOGB-HOG based proposed methodR-HOG based proposed method

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Comparison of performances in each weak classifier

•Error rate of weak classifiers are plotted • If the detection performance is higher with the proposed method, the point is plotted to the right and below the red line

About 95% of weak classifiers improved performance 48

20

30

40

50

20 30 40 50Error before using transition likelihood[%]

B-HOG R-HOG

20

30

50

20 30 40 50Err

or a

fter u

sing

tran

sitio

n lik

elih

ood[%]

Error before using transition likelihood[%]

40

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Implementation on FPGA

49

- Cyclone Ⅲ - Logic cells : 119,088， Memory : 0.48MB

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Conclusion

•For reducing memory usage, HOG feature is binarized • B-HOG feature • R-HOG feature, SR-HOG feature ➡Proposed method reduce memory usage to 1/8, and reduce computational cost to 50% !

•Classifier introducing transition likelihood model based on quantization residual • Weak classifier h(x) predict transition of binary code x ➡Proposed method enables pedestrian detection that is more accurate than that of previous methods !

• Future work • To expand the idea of the proposed method to other learning methods 50