TreeConnect: A Sparse Alternative to Fully Connected Layers

Oliver Richter, Roger Wattenhofer

Distributed Computing Group (DisCo)ETH Zurich

Switzerland 1

“cat”

“dog”

Convolutional Neural Network (CNN)

Fully Connected Layers (FC)

Convolutional Neural Network (CNN)

Fully Connected Layers (FC)

Channel

n 𝜣(n2)m = r n.

n

mhc

1

2

3

= n0.5𝜣(n1.5)

= (n m)0.5. = r n.

c . n hc c

. + c m. = n . r 0.5 nn0.5

. + n0.5 r n. .

L

1 2 3

𝜣(n(2 L - 1) / L). 𝜣(n2)

IMDB sentiment classification

IMDB sentiment classification

Reuters News Categorization

Reuters News Categorization

47,150

25,806

Reuters News Categorization

CIFAR-10 Image Classification

CIFAR-10 Image Classification

CIFAR-10 Image Classification

CIFAR-10 Image Classification

Related Work

Questions & Answers

https://github.com/OliverRichter/TreeConnect

Code?

Further Questions?

richtero@ethz.ch