From Signal to SymbolsApproximate Category Theory and Conceptual Regularization

Anthony Bak

SF Data Mining, November 2015

Disclaimer

Everything in this presentation is stolen (with permission). No ideas, images orformulas are my own excepting errors which are surely mine.

I M. Ovsjanikov, M. Ben-Chen, J. Solomon, A. Butscher and L. Guibas. Functional Maps: A FlexibleRepresentation of Maps Between Shapes. ACM Transactions on Graphics. 31(4), 2012 – Siggraph 2012

I F. Wang, Q. Huang, and L. Guibas. Image Co-Segmentation via Consistent Functional Maps. The 14thInternational Conference on Computer Vision (ICCV). Sydney, Australia, December 2013.

I Q. Huang, F. Wang, L. Guibas, Functional map networks for analyzing and exploring large shapecollections, ACM Transactions on Graphics (TOG), Volume 33, Issue 4, July 2014

I R. Rustamov, D. Romano, A. Reiss, and L. Guibas. Compact and Informative Representation of FunctionalConnectivity for Predictive Modeling. Medical Image Computing and Computer Assisted InterventionConference (MICCAI), 2014

I Syntactic and Functional Variability of a Million Code Submissions in a Machine Learning MOOC, J.Huang, C. Piech, A. Nguyen, L. Guibas. In the 16th International Conference on Artificial Intelligence inEducation (AIED 2013) Workshop on Massive Open Online Courses (MOOCshop) Memphis, TN, USA,July 2013.

Special thanks to Leo Guibas who kindly sent me his images and presentation onthis material. http://geometry.stanford.edu/

Disclaimer

Everything in this presentation is stolen (with permission). No ideas, images orformulas are my own excepting errors which are surely mine.

I M. Ovsjanikov, M. Ben-Chen, J. Solomon, A. Butscher and L. Guibas. Functional Maps: A FlexibleRepresentation of Maps Between Shapes. ACM Transactions on Graphics. 31(4), 2012 – Siggraph 2012

I F. Wang, Q. Huang, and L. Guibas. Image Co-Segmentation via Consistent Functional Maps. The14th International Conference on Computer Vision (ICCV). Sydney, Australia, December 2013.

I Q. Huang, F. Wang, L. Guibas, Functional map networks for analyzing and exploring large shapecollections, ACM Transactions on Graphics (TOG), Volume 33, Issue 4, July 2014

I R. Rustamov, D. Romano, A. Reiss, and L. Guibas. Compact and Informative Representation of FunctionalConnectivity for Predictive Modeling. Medical Image Computing and Computer Assisted InterventionConference (MICCAI), 2014

I Syntactic and Functional Variability of a Million Code Submissions in a Machine Learning MOOC, J.Huang, C. Piech, A. Nguyen, L. Guibas. In the 16th International Conference on Artificial Intelligence inEducation (AIED 2013) Workshop on Massive Open Online Courses (MOOCshop) Memphis, TN, USA,July 2013.

Special thanks to Leo Guibas who kindly sent me his images and presentation onthis material. http://geometry.stanford.edu/

Big Picture

We want to bridge the gap between Human and Computer understanding ofsensor data

I Human understanding and situational reasoning comes from

I Having models of the world consisting of symbols (Cars, words etc.)I Relating sensor input to past experience and the model

Here we present a way to build the symbols from signals by looking for invariantsof the collection. Formally

I We build networks relating signals to each otherI We transport information through the networkI Concepts emerge as fixed points in the network

Big Picture

We want to bridge the gap between Human and Computer understanding ofsensor data

I Human understanding and situational reasoning comes fromI Having models of the world consisting of symbols (Cars, words etc.)

I Relating sensor input to past experience and the model

Here we present a way to build the symbols from signals by looking for invariantsof the collection. Formally

I We build networks relating signals to each otherI We transport information through the networkI Concepts emerge as fixed points in the network

Big Picture

We want to bridge the gap between Human and Computer understanding ofsensor data

I Human understanding and situational reasoning comes fromI Having models of the world consisting of symbols (Cars, words etc.)I Relating sensor input to past experience and the model

Here we present a way to build the symbols from signals by looking for invariantsof the collection. Formally

I We build networks relating signals to each otherI We transport information through the networkI Concepts emerge as fixed points in the network

Big Picture

We want to bridge the gap between Human and Computer understanding ofsensor data

I Human understanding and situational reasoning comes fromI Having models of the world consisting of symbols (Cars, words etc.)I Relating sensor input to past experience and the model

Here we present a way to build the symbols from signals by looking for invariantsof the collection. Formally

I We build networks relating signals to each otherI We transport information through the networkI Concepts emerge as fixed points in the network

Big Picture

We want to bridge the gap between Human and Computer understanding ofsensor data

I Human understanding and situational reasoning comes fromI Having models of the world consisting of symbols (Cars, words etc.)I Relating sensor input to past experience and the model

Here we present a way to build the symbols from signals by looking for invariantsof the collection. Formally

I We build networks relating signals to each other

I We transport information through the networkI Concepts emerge as fixed points in the network

Big Picture

We want to bridge the gap between Human and Computer understanding ofsensor data

I Human understanding and situational reasoning comes fromI Having models of the world consisting of symbols (Cars, words etc.)I Relating sensor input to past experience and the model

Here we present a way to build the symbols from signals by looking for invariantsof the collection. Formally

I We build networks relating signals to each otherI We transport information through the network

I Concepts emerge as fixed points in the network

Big Picture

We want to bridge the gap between Human and Computer understanding ofsensor data

I Human understanding and situational reasoning comes fromI Having models of the world consisting of symbols (Cars, words etc.)I Relating sensor input to past experience and the model

Here we present a way to build the symbols from signals by looking for invariantsof the collection. Formally

I We build networks relating signals to each otherI We transport information through the networkI Concepts emerge as fixed points in the network

Objects and Their FunctionsIn the network nodes will represent objects we are trying to study. Information (orannotations) of our objects will be encoded as functions. For example

I Segmentation or "part indicators"

I Geometric properties eg. Eigenfunctions of Laplace-Beltrami operator,curvature

I Descriptors (eg. SIFT)I etc.

Objects and Their FunctionsIn the network nodes will represent objects we are trying to study. Information (orannotations) of our objects will be encoded as functions. For example

I Segmentation or "part indicators"I Geometric properties eg. Eigenfunctions of Laplace-Beltrami operator,

curvature

I Descriptors (eg. SIFT)I etc.

Objects and Their FunctionsIn the network nodes will represent objects we are trying to study. Information (orannotations) of our objects will be encoded as functions. For example

I Segmentation or "part indicators"I Geometric properties eg. Eigenfunctions of Laplace-Beltrami operator,

curvatureI Descriptors (eg. SIFT)

I etc.

Objects and Their FunctionsIn the network nodes will represent objects we are trying to study. Information (orannotations) of our objects will be encoded as functions. For example

I Segmentation or "part indicators"I Geometric properties eg. Eigenfunctions of Laplace-Beltrami operator,

curvatureI Descriptors (eg. SIFT)I etc.

Information TransportWe will assume real valued functions for the rest of the discussion and write C(P)for the space of real valued functions on P. Given two objects, L, C and a mapφ : L → C.

I We get a map Tφ : C(C)→ C(L) by composition.f ∈ C(C) 7→ f ◦ φ ∈ C(L)

I Tφ is a linear operator.

Information TransportWe will assume real valued functions for the rest of the discussion and write C(P)for the space of real valued functions on P. Given two objects, L, C and a mapφ : L → C.

I We get a map Tφ : C(C)→ C(L) by composition.f ∈ C(C) 7→ f ◦ φ ∈ C(L)

I Tφ is a linear operator.

Information TransportWe will assume real valued functions for the rest of the discussion and write C(P)for the space of real valued functions on P. Given two objects, L, C and a mapφ : L → C.

I We get a map Tφ : C(C)→ C(L) by composition.f ∈ C(C) 7→ f ◦ φ ∈ C(L)

I Tφ is a linear operator.

Information Transport

I Information is transported between objects by applying a linear operatorTCL : C(C)→ C(L).

I We relax the condition that the linear maps are induced from maps on theunderlying objects.

Network Regularization

We want to use the network of relationships between objects to constrain thespace of possible solutions. To that end we require

I The transport of information from C to L does not depend on the path taken.

C(C) C(B)

C(A) C(L)

TCB

TCA TBL

TAL

Network Regularization

We want to use the network of relationships between objects to constrain thespace of possible solutions. To that end we require

I The transport of information from C to L does not depend on the path taken.

C(C) C(B)

C(A) C(L)

TCB

TCA TBL

TAL

General Procedure

To apply this to an exampleI Construct a network consisting of similar objects

I Use transport of similarity measures to fit our linear transformationsI Use these transformations to transport information through the network to

solve some problem

General Procedure

To apply this to an exampleI Construct a network consisting of similar objectsI Use transport of similarity measures to fit our linear transformations

I Use these transformations to transport information through the network tosolve some problem

General Procedure

To apply this to an exampleI Construct a network consisting of similar objectsI Use transport of similarity measures to fit our linear transformationsI Use these transformations to transport information through the network to

solve some problem

Example

Image Co-Segmentation

Problem

Task: Jointly segment a set of related images

I Same object with different viewpoints and scales

I Similar objects of the same class

Images provide weak supervision of each other.

Problem

Task: Jointly segment a set of related imagesI Same object with different viewpoints and scales

I Similar objects of the same class

Images provide weak supervision of each other.

Problem

Task: Jointly segment a set of related imagesI Same object with different viewpoints and scales

I Similar objects of the same class

Images provide weak supervision of each other.

Problem

Task: Jointly segment a set of related imagesI Same object with different viewpoints and scales

I Similar objects of the same class

Images provide weak supervision of each other.

Images to Image Network

We create a (sparse) similarity graph using a gaussian kernel function on GISTimage descriptors for each image. To each edge we assign the weight

wij = exp(−||gi − gj ||2

2σ) σ = median(||gi − gj ||)

We connect each image to its k = 30 most similar neighbors.

Images to Image Network

Image Function Space

Use a super pixel segmentation of the images and build a graph by takingI Nodes are super pixelsI Edges are weighted by length of shared boundary

The function space we associate to each image is the space of real valuedfunctions on this graph.

Hierarchical SubspaceOn a graph have the graph laplacian L = D −W where D is the diagonal degreematrix and W is the edge weight matrix.

I The eigenvectors (eigenfunctions) of L are ordered by "scale". For example, ifthere are k components to the graph then the first k eigenfunctions areindicator functions on the components. The next smallest eigenvector is usedin many graph cut algorithms.Examples of one-dimensional mappings

u2 u3

u4 u8

Radu Horaud Graph Laplacian Tutorial

We choose the subspace spanned by the first 30 eigenvectors. This keep thedimensionality of the problem under control and the hierarchy assures us that thisis reasonable.

Hierarchical SubspaceOn a graph have the graph laplacian L = D −W where D is the diagonal degreematrix and W is the edge weight matrix.

I The eigenvectors (eigenfunctions) of L are ordered by "scale". For example, ifthere are k components to the graph then the first k eigenfunctions areindicator functions on the components. The next smallest eigenvector is usedin many graph cut algorithms.

Examples of one-dimensional mappings

u2 u3

u4 u8

Radu Horaud Graph Laplacian Tutorial

We choose the subspace spanned by the first 30 eigenvectors. This keep thedimensionality of the problem under control and the hierarchy assures us that thisis reasonable.

Hierarchical SubspaceOn a graph have the graph laplacian L = D −W where D is the diagonal degreematrix and W is the edge weight matrix.

I The eigenvectors (eigenfunctions) of L are ordered by "scale". For example, ifthere are k components to the graph then the first k eigenfunctions areindicator functions on the components. The next smallest eigenvector is usedin many graph cut algorithms.

Examples of one-dimensional mappings

u2 u3

u4 u8

Radu Horaud Graph Laplacian Tutorial

We choose the subspace spanned by the first 30 eigenvectors. This keep thedimensionality of the problem under control and the hierarchy assures us that thisis reasonable.

Hierarchical SubspaceOn a graph have the graph laplacian L = D −W where D is the diagonal degreematrix and W is the edge weight matrix.

I The eigenvectors (eigenfunctions) of L are ordered by "scale". For example, ifthere are k components to the graph then the first k eigenfunctions areindicator functions on the components. The next smallest eigenvector is usedin many graph cut algorithms.Examples of one-dimensional mappings

u2 u3

u4 u8

Radu Horaud Graph Laplacian Tutorial

We choose the subspace spanned by the first 30 eigenvectors. This keep thedimensionality of the problem under control and the hierarchy assures us that thisis reasonable.

Hierarchical SubspaceOn a graph have the graph laplacian L = D −W where D is the diagonal degreematrix and W is the edge weight matrix.

I The eigenvectors (eigenfunctions) of L are ordered by "scale". For example, ifthere are k components to the graph then the first k eigenfunctions areindicator functions on the components. The next smallest eigenvector is usedin many graph cut algorithms.Examples of one-dimensional mappings

u2 u3

u4 u8

Radu Horaud Graph Laplacian Tutorial

We choose the subspace spanned by the first 30 eigenvectors. This keep thedimensionality of the problem under control and the hierarchy assures us that thisis reasonable.

Eigenfunctions on Images

Joint Estimation of Transfer Functions

We do joint estimation of the transfer functions by optimizing over transfer matricesTij

I Aligning image features associated with each super pixel (e.g.. average RGBcolor)

I A regularization term that penalizes mapping eigenspaces with very differenteigenvalues to each other

I A cycle consistency term

This is solvable. So we get our consistent maps Tij

Joint Estimation of Transfer Functions

We do joint estimation of the transfer functions by optimizing over transfer matricesTij

I Aligning image features associated with each super pixel (e.g.. average RGBcolor)

I A regularization term that penalizes mapping eigenspaces with very differenteigenvalues to each other

I A cycle consistency termThis is solvable. So we get our consistent maps Tij

Segmentation

The actual segmentation is done by finding

I Best cut functionI Subject to the consistency between cut functions on other images

This is a joint optimization problem and also solvable

.

Segmentation

The actual segmentation is done by finding

I Best cut functionI Subject to the consistency between cut functions on other images

This is a joint optimization problem and also solvable.

Experimental Results

Tested on three standard datasetsI iCoseg

I Very similar or the same object in each classI 5-10 images per class

I MSRCI Very similar objects in each classI 30 images per class

I PascalI Larger scale and variability

Experimental Results: iCoseg

Vicente is a supervised method

Experimental Results: iCoseg

Vicente is a supervised method

iCoseg: 5 images per class are shown

iCoseg: 5 images per class are shown

iCoseg: 5 images per class are shown

iCoseg: 5 images per class are shown

MSRC: 5 images per class are shown

MSRC: 5 images per class are shown

PASCAL: 10 images per class are shown

PASCAL: 10 images per class are shown

PASCAL: 10 images per class are shown

PASCAL: 10 images per class are shown

The Network is the Abstraction

Plato’s cow

Summary

Classical view:I A hierarchy moves you from signals, to parts, to symbols,...I "Vertical"

Alternate view:I Symbols emerge from the network of signal relationships as invariants.I "Horizontal"