Towards Neural Homology Theory - wgusswguss.ml/public/pdf/towards_neural_homology_theory.pdf ·...

Towards Neural Homology Theory

William H. Guss 1,2 Ruslan Salakhutdinov 1

{wguss, rsalakhu}@cs.cmu.edu

1Carnegie Mellon University

2Machine Learning at Berkeley

February 1, 2018

Guss & Salakhutdinov (CMU) Neural Homology Theory February 1, 2018 1 / 40

The problem of architecture selection

Deep learning has been extremely successful, in part due to the eliminationof feature engineering.

Research focus has shifted: best features Ñ best architectures.

A1 A2

Figure: Examples of two competing convolutional architecture A1 and A2.

In computer vision, for example, a large body of work focuses onimproving the initial architectural choices of AlexNet.Architectures are becoming hyperspecialized.


The problem of architecture selection

Deep learning has been extremely successful, in part due to the eliminationof feature engineering.

Research focus has shifted: best features Ñ best architectures.

A1 A2

Figure: Examples of two competing convolutional architecture A1 and A2.

In computer vision, for example, a large body of work focuses onimproving the initial architectural choices of AlexNet.

Architectures are becoming hyperspecialized.


The problem of architecture selection (cont.)

Despite the success of this approach, there are still not generalprinciples for choosing architectures in arbitrary settings.

To scale deep learning to new problems without expert architecturedesigners, the problem of architecture selection must be understood.

Figure: Graduate student ascent on the ImageNet dataset by year.


A partial solution: neural expressivity theory

Neural expressivity theory: properties of neural networks Ñ expressivityand generalization capability ([RPK`16], [DFS16], [Gus16]).

E.g. shallow networks need exponentially many more units to expresscertain functions than deeper ones; deep networks can express curvaturemore efficiently, etc.

This can only be used to determine an architecture in practice if itis understood how expressive a model need be in order to solve a

problem


Our approach: data-first architecture selection

Given a learning problem (some dataset), which architectures are suitablyregularized and expressive enough to learn and generalize on that problem?

DAS: develop some objective measure of data complexity, and thencharacterize neural architectures by their ability to learn subject to thatcomplexity.


Towards a measure of datacomplexity


A brief introduction to topology

Topology characterizes shapes, spaces, and sets by their connectivity.

Def (Homeomorphism). Informally, wesay that two topological spaces A andB are equivalent (A – B) if there is acontinuous function f : AÑ B that hasan inverse f ´1 that is also continuous.

Topology can differentiate sets in ameaningful way that discards certainirrelevant properties such as rotation, translation, curvature, etc.


Topology differentiates datasets

In this view D1 fl D2. Furthermore h12 cannot express decision boundarieswith the topology of D2.

We can characterize architectures by topological expressivity.


A brief introduction to algebraic topology

Algebraic topology lets us explicitly compute which spaces are equivalentto eachother.

Figure: An illustration of the core philosophy of algebraic topology: ’functoraly’reduce hard problems in topology to easy ones in group theory.


Homology: a tool for computing topology

Def (Homology). If X is a topological space,

HnpX q “ Zβn is called the nth homology group of X .

βn is the number of ’holes’ of dimension n in X .

H0pD1q “ Z2

H1pD1q “ t0uH2pD1q “ t0u

H0pD2q “ Z4

H1pD2q “ Z2

H2pD2q “ t0u

HpD1q ď HpD2q and D2 requires more complex architectures


Homology: a tool for computing topology

Def (Homology). If X is a topological space, then

HnpX q “ Zβn is called the nth homology group of X if βn is thenumber of ’holes’ of dimension n in X .

the homology of X is defined as HpX q “ tHnpX qu8n“0.

Theorem (Bredon). If X – Y then HpX q “ HpY q.


Homological Characterization ofNeural Architectures


The power of homological characterization

Homology is a stringent measure for characterizing neural architectures:

Let M Ă X be a topological manifold.

Let HSpf q be the homology of the support off , i.e. HSpf q “ Hptx : f pxq ą 0uq.

Theorem (The Homological Principle ofGeneralization). If for all f P F ,HSpf q ‰ HpMq, then there exists A Ă Xsuch that f missclassifies A.

Figure: When homologycannot be expressed, thereexists a misclassified set.


The power of homological characterization

Homology is a stringent measure for characterizing neural architectures:

Let M Ă X be a topological manifold.

Let HSpf q be the homology of the support off , i.e. HSpf q “ Hptx : f pxq ą 0uq.

Theorem (The Homological Principle ofGeneralization). If for all f P F ,HSpf q ‰ HpMq, then there exists A Ă Xsuch that f missclassifies A. Figure: When homology

cannot be expressed, thereexists a misclassified set.


Architecture selection in the lense of topology

Given a dataset M, for which architectures A does there exista neural network f P FA such that HSpf q “ HpMq?


An empirical approach

We’ll first try and answer this question empirically.


An empirical approach: Synthetic data


1 Generate data of increasing homological complexity.


An empirical approach: Persistent homology



2 Compute the persistent homology of the synthetic data.

Figure: Left: The local disconnectedness of datasets prevents directcomputation of their homology. Right: An illustration of computingpersistent homology on a collection of points ([TZH15])


An empirical approach: Persistent homology



2 (Sanity check) Real data is homologically rich!

Figure: The persistent homology barcodes of classes in the CIFAR-10 andUCI Protein Localization Datasets.


An empirical approach: Characterization




3 Train models of increasing architectural complexity.


Empirical results: Topological phase transitions

Figure: Average testing error of different architectures on datasets of increasinghomological complexity.


Empirical results: Homological characterization

There are integral relationshipsbetween homological complexityand learnable architectures.

Eg.

H0pDq “ Zm ùñ

there exists a NN with h “ m ` 2that converges to zero error D.

H0pDq “ Zm and H1pDq “ Z ùñ

the same holds with h ě 3m ´ 1.


Empirical results: Homological characterization

There are integral relationshipsbetween homological complexityand learnable architectures.

Eg. H0pDq “ Zm ùñ

there exists a NN with h “ m ` 2that converges to zero error D.

H0pDq “ Zm and H1pDq “ Z ùñ

the same holds with h ě 3m ´ 1.


Neural Homology Theory


Towards a complete neural homology theory


Neural Homology Theory.


Neural networks as set expressions.

Neural networks with ReLu-like functions can be converted into setexpression.

1 The decision region of a single hidden unit is a halfspace.

Figure: An illustration of halfspaces induced by ReLu-like networks.




1 The decision region of a single hidden unit is a halfspace.2 The decision region of deeper hidden units applies X,Y, to

shallower decision regions.

tx : hcpxq ą 0u “

paX bq Y p aX bq Y paX bq

“ aY b






tx : hcpxq ą 0u “ paX bq

Y p aX bq Y paX bq

“ aY b






tx : hcpxq ą 0u “ paX bq Y p aX bq

Y paX bq

“ aY b






tx : hcpxq ą 0u “ paX bq Y p aX bq Y paX bq

“ aY b




1 The decision region of a single hidden unit is a halfspace.

2 The decision region of deeper hidden units applies X,Y, toshallower decision regions.

Formally:

tx : Npxq ą 0u “ď

γPSpW q

č

hiPhidden

tx : signpγi qhi pxq ą 0u

with Npxq “ ReLupW Thpxqq and

SpW q “ th : W Th ą 0u X t´1, 1u|hidden|.


Homology of set algebra on halfspaces

The homology of decision regions is computed with tools forcomputing homology on set expressions.

1 Notation. For half spaces a, b denote

§ The union a` b :“ aY b§ The intersection ab :“ aX b§ The ”sum” of homologies Hpaq ‘ Hpbq§ The ”subtraction of homologies” Hpaq{Hpbq.§ For the rest of the talk we’ll drop H; a :“ Hpaq.

For example,

# holes “a‘ b

ab ` c ‘ d:“

Hpaq ‘ Hpbq

HpaX b Y cq ‘ Hpdq




1 Notation. For half spaces a, b denote§ The union a` b :“ aY b

§ The intersection ab :“ aX b§ The ”sum” of homologies Hpaq ‘ Hpbq§ The ”subtraction of homologies” Hpaq{Hpbq.§ For the rest of the talk we’ll drop H; a :“ Hpaq.

For example,

# holes “a‘ b

ab ` c ‘ d:“

Hpaq ‘ Hpbq





1 Notation. For half spaces a, b denote§ The union a` b :“ aY b§ The intersection ab :“ aX b

§ The ”sum” of homologies Hpaq ‘ Hpbq§ The ”subtraction of homologies” Hpaq{Hpbq.§ For the rest of the talk we’ll drop H; a :“ Hpaq.

For example,

# holes “a‘ b

ab ` c ‘ d:“

Hpaq ‘ Hpbq





1 Notation. For half spaces a, b denote§ The union a` b :“ aY b§ The intersection ab :“ aX b§ The ”sum” of homologies Hpaq ‘ Hpbq

§ The ”subtraction of homologies” Hpaq{Hpbq.§ For the rest of the talk we’ll drop H; a :“ Hpaq.

For example,

# holes “a‘ b

ab ` c ‘ d:“

Hpaq ‘ Hpbq





1 Notation. For half spaces a, b denote§ The union a` b :“ aY b§ The intersection ab :“ aX b§ The ”sum” of homologies Hpaq ‘ Hpbq§ The ”subtraction of homologies” Hpaq{Hpbq.

§ For the rest of the talk we’ll drop H; a :“ Hpaq.

For example,

# holes “a‘ b

ab ` c ‘ d:“

Hpaq ‘ Hpbq





1 Notation. For half spaces a, b denote§ The union a` b :“ aY b§ The intersection ab :“ aX b§ The ”sum” of homologies Hpaq ‘ Hpbq§ The ”subtraction of homologies” Hpaq{Hpbq.§ For the rest of the talk we’ll drop H; a :“ Hpaq.

For example,

# holes “a‘ b

ab ` c ‘ d:“

Hpaq ‘ Hpbq





1 Notation.2 Neural Cotorsion Algebra. We consider H0ptx : Npxq ą 0uq

§ For any halfspace a, we know a “ 1.§ Rule #1: a‘ b “ ab ‘ pa` bq.§ Rule #2: If c “ a‘ b then a “ c{b.§ Rule #3: Normal set operations work: apb` cq “ ab` ac , aabc “ abc.

Example. Let a, b, c be halfspaces

H0paX b Y cq “

ab ` c

“ab ‘ c

abc“

a‘ b ‘ c

abc ‘ a` b

« 3´ H0paX b X cq ` H0paY bq.





§ For any halfspace a, we know a “ 1.

§ Rule #1: a‘ b “ ab ‘ pa` bq.§ Rule #2: If c “ a‘ b then a “ c{b.§ Rule #3: Normal set operations work: apb` cq “ ab` ac , aabc “ abc.


H0paX b Y cq “

ab ` c

“ab ‘ c

abc“

a‘ b ‘ c

abc ‘ a` b






§ For any halfspace a, we know a “ 1.§ Rule #1: a‘ b “ ab ‘ pa` bq.

§ Rule #2: If c “ a‘ b then a “ c{b.§ Rule #3: Normal set operations work: apb` cq “ ab` ac , aabc “ abc.


H0paX b Y cq “

ab ` c

“ab ‘ c

abc“

a‘ b ‘ c

abc ‘ a` b






§ For any halfspace a, we know a “ 1.§ Rule #1: a‘ b “ ab ‘ pa` bq.§ Rule #2: If c “ a‘ b then a “ c{b.

§ Rule #3: Normal set operations work: apb` cq “ ab` ac , aabc “ abc.


H0paX b Y cq “

ab ` c

“ab ‘ c

abc“

a‘ b ‘ c

abc ‘ a` b








H0paX b Y cq “

ab ` c

“ab ‘ c

abc“

a‘ b ‘ c

abc ‘ a` b








H0paX b Y cq “ ab ` c

“ab ‘ c

abc“

a‘ b ‘ c

abc ‘ a` b









“ab ‘ c

abc

“a‘ b ‘ c

abc ‘ a` b









“ab ‘ c

abc“

a‘ b ‘ c

abc ‘ a` b



Neural homology theory for architecture selection.

Neural homology theory lets us answer questions in architectureselection.

Example. Can a 3 hidden unit neural network generalize on a dataset withH0pMq “ 10?

Neural homology theory says no.



Theorem

Any 3 neuron ReLu network fails on data with ě 4 clusters.

Proof.

Let h3pxq “ ReLupwT phapxq, hbpxq, hcpxqqq.

1 [NN Ñ Set Algebra] h3 ą 0 “ aY b Y c.

2 [Set Algebra Ñ Homology]

H0ph3 ą 0q “ a` b ` c

“a` b ‘ c

pa` bqc“

a` b ‘ c

ac ` bc

“a` b ‘ c“

ac‘bcacbc

‰ “a` b ‘ c“

ac‘bcabc

‰ .



Theorem


Proof.




H0ph3 ą 0q “ a` b ` c

“a` b ‘ c

pa` bqc

“a` b ‘ c

ac ` bc

“a` b ‘ c“

ac‘bcacbc

‰ “a` b ‘ c“

ac‘bcabc

‰ .



Theorem


Proof.




H0ph3 ą 0q “ a` b ` c

“a` b ‘ c

pa` bqc“

a` b ‘ c

ac ` bc

“a` b ‘ c“

ac‘bcacbc

‰ “a` b ‘ c“

ac‘bcabc

‰ .



Theorem


Proof.




H0ph3 ą 0q “ a` b ` c

“a` b ‘ c

pa` bqc“

a` b ‘ c

ac ` bc

“a` b ‘ c“

ac‘bcacbc

‰

“a` b ‘ c“

ac‘bcabc

‰ .



Theorem


Proof.




H0ph3 ą 0q “ a` b ` c

“a` b ‘ c

pa` bqc“

a` b ‘ c

ac ` bc

“a` b ‘ c“

ac‘bcacbc

‰ “a` b ‘ c“

ac‘bcabc

‰ .



Theorem


Proof.


1 Then applying algebra

a` b ‘ c“

ac‘bcabc

‰ ď a` b ‘ c ď 3.

2 Therefore# Regions “ H0ph3 ą 0q ď 3.


Remarks & Questions


[Bre13] Glen E Bredon. Topology and geometry, volume 139. SpringerScience & Business Media, 2013.

[DFS16] Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeperunderstanding of neural networks: The power of initializationand a dual view on expressivity. In Advances In NeuralInformation Processing Systems, pages 2253–2261, 2016.

[RPK`16] Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, andJascha Sohl-Dickstein. On the expressive power of deep neuralnetworks. arXiv preprint arXiv:1606.05336, 2016.

[TZH15] Chad M Topaz, Lori Ziegelmeier, and Tom Halverson.Topological data analysis of biological aggregation models.PloS one, 10(5):e0126383, 2015.


Appendix


Homology

Definition (Homology Theory, [Bre13])

A homology theory on the on Top2 is a function H : Top2 Ñ Ab assigningto each pair pX ,Aq of spaces a graded (abelian) group tHppX ,Aqu, and toeach map f : pX ,Aq Ñ pY ,Bq, homomorphismsf˚ : HppX ,Aq Ñ HppY ,Bq, together with a natural transformation offunctors B˚ : HppX ,Aq Ñ Hp´1pX ,Aq, called the connectinghomomorphism (where we use H˚pAq to denote H˚pA,Hq) such that thefollowing five axioms are satisfied.


Homology

Definition (Homology Theory, [Bre13])

1 If f » g : pX ,Aq Ñ pY ,Bq then f˚ “ g˚ : H˚pX ,Aq Ñ H˚pY ,Bq.

2 For the inclusions i : AÑ X and j : X Ñ pX ,Aq the sequencesequence of inclusions and connecting homomorphisms are exact.

3 Given the pair pX ,Aq and an open set U Ă X such thatclpUq Ă intpAq then the inclusion k : pX ´ U,A´ Uq Ñ pX ,Aqinduces an isomorphism k˚ : H˚pX ´ U,A´ Uq Ñ H˚pX ,Aq

4 For a one point space P,Hi pPq “ 0 for all i ‰ 0.

5 For a topological sum X “ `αXα the homomorphism

à

piαq˚ :à

HnpXαq Ñ HnpX q

is an isomorphism, where iα : Xα Ñ X is the inclusion.


Towards Neural Homology Theory - wgusswguss.ml/public/pdf/towards_neural_homology_theory.pdf ·...

Documents

Transcript of Towards Neural Homology Theory - wgusswguss.ml/public/pdf/towards_neural_homology_theory.pdf ·...