A Dirichlet Process Mixture Model for Spherical Data

Julian Straub, Jason Chang, Oren Freifeld, John W. Fisher IIIComputer Science and Artificial Intelligence Laboratory

Massachusetts Institute of Technology{jstraub, jchang7, freifeld, fisher}@csail.mit.edu

Abstract

Directional data, naturally represented aspoints on the unit sphere, appear in manyapplications. However, unlike the case of Eu-clidean data, flexible mixture models on thesphere that can capture correlations, handlean unknown number of components and ex-tend readily to high-dimensional data haveyet to be suggested. For this purpose wepropose a Dirichlet process mixture modelof Gaussian distributions in distinct tangentspaces (DP-TGMM) to the sphere. Impor-tantly, the formulation of the proposed modelallows the extension of recent advances in ef-ficient inference for Bayesian nonparametricmodels to the spherical domain. Experimentson synthetic data as well as real-world 3Dsurface normal and 20-dimensional semanticword vector data confirm the expressivenessand applicability of the DP-TGMM.

1 Introduction

Many applications of interest involve measurements ofdirectional data. In 3D scenes, unit-length surface nor-mals extracted from point clouds [17, 24, 44] reside ina 2D manifold (i.e., the unit sphere in R3). In biol-ogy, protein backbone measurements are described andclassified based on their angular configurations in theso-called Ramachandran plots [37]. Directional dataalso exists outside of the 3D world. E.g., the wordscounts in a corpus of documents can be viewed as di-rectional data once normalized to have unit `2 norm.Word-frequency vectors are often clustered using thecosine similarity [10], which measures the cosine of the

Appearing in Proceedings of the 18th International Con-ference on Artificial Intelligence and Statistics (AISTATS)2015, San Diego, CA, USA. JMLR: W&CP volume 38.Copyright 2015 by the authors.

π α ∆, ν

zi xi Σk

xi µk S2

N (0,Σk)N (0,Σj)

TµkS2TµjS2NNNNNNN N∞

Figure 1: The graphical model of the proposed Dirich-let process tangential Gaussian mixture model (DP-TGMM) and an illustrative drawing for K = 2 clus-ters, k and j, in their respective tangent spaces to thesphere S2. For more details refer to Sec. 3.2.

angle formed by two vectors. This measure essentiallytreats the word-frequency vectors as directional data,and has been shown to be superior to Euclidean dis-tance for document clustering [46]. Another exampleof directional data is semantic word vectors [32], whichassociate a high-dimensional vector with each word ina given corpus. The semantic word vectors capture thesemantic context of the associated words, and shouldnot to be confused with the word-frequency vectorsof documents. Again, cosine similarity is used as thedistance measure to find words with similar meaning.

One common task in many of these applications is togroup the data into similar clusters. Due to the non-linearity of the hyper-sphere, clustering on the spher-ical manifold is often treated in an ad-hoc mannerby either ignoring the geometry of the sphere or us-ing overly-restricted models. In this work, we presenta flexible Bayesian nonparametric (BNP) model fordata residing on a hyper-sphere that respects the in-herent geometry of the manifold. As shown in Fig. 1,our approach draws on the Dirichlet process Gaussianmixture model (DP-GMM), and models full covariancematrices on (linear) tangent spaces to the sphere, asopposed to the isotropic covariances associated with avon-Mises-Fisher (vMF) distribution [1, 2, 38, 49]. Im-portantly, the covariances of the Gaussians, capturingintra-cluster correlations, have analytical conjugate

A Dirichlet Process Mixture Model for Spherical Data

Table 1: Properties of different typically-used clustering algorithms for directional data.

k-means spkm vMF-MM TGMM DP-vMF-MM DP-GMM DP-TGMM[21] [10, 49] [1] [12, 42] [2] [6] (proposed)

Spherical geometry · X X X X · XBayesian inference · · X X X X XAnisotropic covariance · · · X · X XBayesian nonparametric · · · · X X XParallelizeable X X X X · X X

priors that enable efficient inference. Additionally,the approach transparently scales to high-dimensionaldata. We extend the efficient inference method of [6], aparallelized restricted Gibbs sampler with sub-clustersplit/merge moves, to account for the geometry of thesphere. Moreover, we show how to combine sufficientstatistics from tangent spaces around different pointsof tangency to propose merges efficiently.

To highlight the differences of the proposed Dirich-let process tangential Gaussian mixture model (DP-TGMM), we quantitatively compare it with four othermethods on synthetic directional data with ground-truth labels. We demonstrate the scalability and effi-ciency of the inference algorithm as well as the applica-bility of the DP-TGMM to real-world directional databy modeling 3D surface normals extracted from pointclouds. Furthermore, we show its scalability to higherdimensions by clustering the 20-dimensional semanticword vectors of 41k words extracted from the EnglishWikipedia corpus [48].

2 Related Work

We now discuss relevant work related to clustering di-rectional data on a hyper-sphere. Many directionaldistributions exist (e.g., [3, 28, 31]). However, we fo-cus our discussion on work using the von-Mises-Fisher(vMF) distribution [14] due to its popularity.

Several algorithms model directional data using a fi-nite mixture of vMF distributions. Banerjee et al. [1]perform Expectation Maximization (EM) for a finitevMF mixture model to cluster text and genomic data.In the limit, when the vMF concentration parame-ter approaches infinity, this method simplifies to thespherical k-means (spkm) algorithm [10]. Zhong [49]extends the spkm algorithm to online clustering.

The vMF distribution has also been used in BNP mix-ture models. Bangert et al. [2] formulate a Dirichletprocess (DP) [13] vMF mixture model. Their infer-ence relies heavily on the conjugacy of the prior ofthe vMF mean and is difficult to generalize to non-conjugate priors. Furthermore, scaling this methodto large datasets is problematic because the inferenceprocedure is based on the Chinese Restaurant Pro-

cess [36], which cannot be parallelized. Reisinger etal. [38] formulate a finite latent Dirichlet allocationmodel [5] for directional data using vMF distributionswith fixed vMF concentration parameters. This is akinto using a GMM with a fixed variance, which is knownto perform poorly if the model variance does not matchthe noise characteristics.

In general, using vMF distributions has two majorflaws. First, the lack of a closed-form conjugate priorfor the concentration parameter in the vMF distribu-tion complicates posterior inference. Slice samplingmethods [2] partially address this issue, but at the costof extra computation. Often, the concentration pa-rameter is still arbitrarily fixed and not inferred fromthe data (e.g., [38]). More importantly though, thevMF distribution is isotropic. That is, similar to aspherical Gaussian distribution, a vMF distributioncannot capture different variances in each dimension ofthe data or correlations between the dimensions. Wenote that the Fisher-Bingham distribution [28] gen-eralizes the vMF distribution to anisotropic (i.e., el-liptical) distributions on the sphere. While Peel etal. [34] propose an EM-based inference for finite mix-tures of Fisher-Bingham distributions in 3D, exten-sions to higher dimensions are difficult due to the nor-malizer of the probability density function.

In other applications, the inherent geometry of theproblem is ignored and, without taking the spheri-cal geometry into account, algorithms developed forEuclidean geometry are used; e.g., k-means [21], thefinite Gaussian mixture model (GMM) [4], as well asthe Dirichlet process GMM [13, 30].

The work in protein-configuration modeling from Ra-machandran plots [37] exemplifies this well. First Dahlet al. [9] introduced modeling the angular data as aDP-GMM, ignoring the spherical manifold of the an-gular data. To solve this issue Lennox et al. [29] modelthe data on the 3D sphere as a DP-vMF mixture, butrequire an approximation to the vMF posterior. Workby Ting et al. [47] uses an HDP with normal-inverse-Wishart base measure to share data between proteins,but does not respect the manifold of the data.

Approaches utilizing a single tangent space to definedistributions over the hyper-sphere have been pro-

Julian Straub, Jason Chang, Oren Freifeld, John W. Fisher III

posed for rotation estimation and tracking [8, 19]. Fi-nite mixture models of Gaussians in separate tangentspaces have been explored to estimate rigid-body mo-tion in robotics [12] and for human body-pose regres-sion [42]. While Simo-Serra et al. [42] show a way toreduce (but not increase) the number of clusters withintheir EM inference framework, both models are finitemixture models in contrast to our DP-based infinitemixture model.

In contrast to previous approaches, the proposed DP-TGMM allows for anisotropic distributions on thesphere, lends itself to consistent Bayesian inference,and adapts the model complexity to the observations.We develop a corresponding inference algorithm thatcan be parallelized and respects the geometry of theunit sphere. Table 1 highlights the differences betweenthe proposed and previous approaches.

3 BNP Mixtures of Spherical Data

Classical Statistics rely on the Euclidean structureof RD. Thus, due to the nonlinearity of the sphere,the statistical analysis of spherical data requires spe-cial care [15, 31]. In this section we introduce theDirichlet process tangential Gaussian mixture model(DP-TGMM), a mixture model for data lying on theunit sphere, SD−1 = {x : xTx = 1 ; x ∈ RD}. Im-portantly, the model (as well as the inference algo-rithm; cf. Sec. 4) respects that the sphere is a (D−1)-dimensional nonlinear Riemannian manifold. Beforeintroducing the probabilistic model, we now give abrief introduction to the geometric concepts used inthe DP-TGMM. The interested reader can consult [11]for a more detailed discussion.

While SD−1 is nonlinear, every point, p ∈ SD−1, is as-sociated with a linear tangent space, denoted TpSD−1:

TpSD−1 , {x : pT x = 0} . (1)

Elements of TpSD−1 are called tangent vectors andmay be viewed as “arrows” based at p and tangentto SD−1. Note that dim(TpSD−1) = D − 1 and thatthe point of tangency, p, may be identified with theorigin of TpSD−1 (i.e., a zero-length tangent vector).

Due to their linearity, tangent spaces often providea convenient way to model spherical data. In fact,this is also true for more general manifolds [16, 23, 35,43]. This linearity, together with mappings betweenSD−1 and TpSD−1 (cf. Sec. 3.1), enables the modelingand clustering of data points via a zero-mean Gaussiandistribution in a cluster-dependent tangent space. Weillustrate this model in Fig. 2, where x denotes thepoint x ∈ SD−1 mapped to TpSD−1.

TpS2

S2

p

x

xN (0,Σ)

Figure 2: Left: The blue plane illustrates TpS2, thetangent space to the sphere S2 at p ∈ S2. A tangentvector x ∈ TpS2 is mapped to x ∈ S2 via Expp. Right:We describe the data as zero-mean Gaussian in TµS2.

3.1 Geometric Properties of SD−1 & TpSD−1

In statistical modeling, distances are of paramount im-portance. On SD−1, rather than using the Euclideandistance of the ambient space, RD, an appropriatemeasure is the geodesic distance between points, whichis simply the angle between them:

dG : (p, q) 7→ arccos(pT q) , (2)

where p, q ∈ SD−1. The probability measure we willdefine on SD−1 will exploit this distance measure. Apoint, x ∈ SD−1 \ {−p}, is mapped to a point, x ∈TpSD−1, via the Riemannian logarithm:

Logp : x 7→ x = (x− p cos θ) θsin θ (with 0

sin 0 = 1) (3)

where θ = dG(p, x). Conversely, x ∈ TpSD−1 ismapped to x ∈ SD−1 by the Riemannian exponential:

Expp : x 7→ x = p cos(||x||2) + x||x||2 sin(||x||2) . (4)

The `2 norm ||x||2 in TpSD−1 is equal to the distancebetween p and x: ||x||2 = θ = dG(p, x). However, thisis true only since p is the point of tangency. In gen-eral, the distance between two other points in TpSD−1

is not equal to the geodesic distance between their cor-responding points in SD−1. Thus, while a single zero-mean Gaussian in the tangent space around a point,p, provides an effective model for the within-clusterdeviations from p (provided it is the Karcher meanof this cluster – see below), using a Gaussian mixturemodel whose (non-zero mean) components live on thesame tangent space is a poor choice. Thus, in theproposed model, each mixture component exists in itsown tangent space, and each tangent space is uniquewith certainty due to the continuous base measure.

Of special importance is the issue of selecting points oftangency. In this context we utilize the notion of the(so-called1) Karcher mean [20, 26], which generalizes

1See a recent discussion by Karcher [27].

A Dirichlet Process Mixture Model for Spherical Data

the notion of the sample mean from RD to Riemannianmanifolds. Particularly for SD−1, it is defined as alocal minimizer of the following function:

〈x〉 = arg minp∈SD−1

∑N

i=1d2G(p, xi) (5)

where dG(·, ·) is given by Eq. (2). In practice, an iter-ative approach [35] (also included in our supplementalmaterial) efficiently computes the Karcher mean.

3.2 Probabilistic DPMM for Spherical Data

The well known Dirichlet process [13] has been exten-sively used to model data in Euclidean spaces. A DPmixture model (DPMM) uses the DP as a prior toweight a countably-infinite set of clusters, where thedistribution of weights is controlled by a concentrationparameter, α. Here, we formulate the Dirichlet pro-cess tangential Gaussian mixture model (DP-TGMM)which extends DPMMs to data on the unit sphere,SD−1, in a manner that explicitly respects the intrinsicgeometry. The graphical model is depicted in Fig. 1.

The generative DPMM first samples the infinite-lengthcluster proportions, π, from a stick-breaking process[41]. Then cluster assignments, z = {zi}Ni=1, are sam-pled from the categorical distribution defined by π:

π ∼ GEM(1, α) , zi ∼ Cat(π1, π2, . . . ) . (6)

Associated with each cluster, k ∈ {1, . . . ,∞}, is amean location on the sphere, µk, and a covariance, Σk,in the corresponding tangent space, TµkSD−1. Theseparameters are drawn from the following priors:

µk ∼ Unif(SD−1) , Σk ∼ IW(∆, ν) , (7)

where Unif and IW are the uniform and inverse-Wishart distributions, respectively. Note that SD−1

has a finite surface area which is used as the normaliz-ing constant of Unif. This will later play a role in theinference procedure.

An observation, xi, is drawn by sampling from a zero-mean Gaussian with covariance Σk in its correspond-ing tangent space, TµkSD−1, followed by mapping thepoint to SD−1 via the exponential map (Eq. (4)):

xi ∼ Expµzi(N (0,Σzi)) ∀i ∈ {1, . . . , N} ; (8)

see Fig. 2. The Gaussian on TµziSD−1 induces a prob-

ability measure on SD−1. Note this statement is validdespite the fact that the Gaussian has infinite supportin TµziS

D−1 while the sphere is compact and the factthat Expµzi

is not injective (note, however, that it is

injective when restricted to Logµzi(SD−1 \ {p})).

We now describe efficient MCMC inference for afore-mentioned geometry-respecting model.

4 Manifold-Aware MCMC Inference

Markov chain Monte Carlo (MCMC) techniques [39]provide a computational mechanism for sampling fromcomplex Bayesian models. Unfortunately, in DP mix-ture models, MCMC methods are often slow. Whenparameters are marginalized, inference scales poorlybecause algorithms cannot be parallelized. When pa-rameters are instantiated, the algorithm is paralleliz-able, but typically requires approximations and ex-hibits slow convergence. The recent DP sub-clusteralgorithm of [6] addresses these issues by combiningMetropolis-Hastings (MH) split/merge moves with arestricted Gibbs sampler, which is not allowed to addor remove clusters. The resulting Markov chain isguaranteed to converge to the desired posterior distri-bution. Additionally, this approach allows paralleliza-tion and the support of non-conjugate priors.

The DP sub-cluster algorithm proposes splits effec-tively via the MH framework [22] by exploiting an in-ferred auxiliary two-component, “sub-cluster” modelfor each regular cluster. The sub-clusters are inferredwithin the restricted Gibbs sampler. Excluding thevarying complexity of posterior parameter sampling(O(KD2) for a GMM), the computational complexityper MCMC iteration is O(NK + K2), K is the max-imum number of non-empty clusters. While the algo-rithm from [6] was originally suggested for DP modelsin RD, we show here that it can be extended to theDP-TGMM. This extension requires: (1) respectingthe geometry of SD−1 when computing posterior dis-tributions; and (2) combining sufficient statistics fromdifferent tangent spaces to propose splits efficiently.Additionally, we propose a MH algorithm to samplefrom the true posterior of the mean location on SD−1.

4.1 Restricted Gibbs Sampling

We now discuss restricted Gibbs sampling of the la-bels z , {zi}Ni=1, means µ , {µk}Kk=1 and covariancesΣ , {Σk}Kk=1 for K clusters. We note that each Σk isdefined over a separate tangent space, TµkSD−1.

The covariances for each cluster are first sampled.Conditioned on the mean, µk, the data, x , {xi}Ni=1,are modeled via a zero-mean Gaussian distribution inthe tangent plane, TµkSD−1, as defined in Eq. (8).Hence, the same analysis as in the Euclidean spaceapplies, and we sample Σ from the IW posterior [18]:

Σk ∼ p(Σk|x, z, µk) = IW(∆ + Sk, ν +Nk) (9)

where Ik , {i : zi = k} is the set of indices with labelk, Nk , |Ik| counts the points assigned to cluster k,and Sµk is the scatter matrix at TµkSD−1, defined as:

Sµk ,∑

i∈IkLogµk(xi)Logµk(xi)

T . (10)

Julian Straub, Jason Chang, Oren Freifeld, John W. Fisher III

Note, however, that the geometry of SD−1 ren-ders the frequently-required computation of Sµk in-efficient. The bottleneck of the calculation is at-tributed to the computationally-intensive evaluationof {Logµk(xi)}i∈Ik that depends on the point of tan-gency, µk, which constantly changes during inference.

To circumvent this issue, we exploit the fact thatLogµk(xi) ≈ Logµk(〈x〉k) + Log〈x〉k(xi), and make thefollowing approximation for the scatter matrix:

Sµk ≈ S〈x〉k +NkLogµk(〈x〉k)Logµk(〈x〉k)T , (11)

where 〈x〉k is the Karcher mean of xIk and S〈x〉k is thescatter matrix computed in the tangent plane of 〈x〉k.This approximation is more efficient because the com-putation of S〈x〉k can be reused when µk changes. Wewill also use this approximation for proposing merges.

Conditioned on the sampled covariance matrix, Σk,we then sample µk. Ideally, we would sample directlyfrom the following posterior distribution of µk:

p(µk|x, z,Σk) ∝ p(µk)p(x|µk, z,Σk) (12)

= p(µk)∏

i∈IkN (Logµk(xi); 0,Σk).

Unfortunately, due to the nonlinearity of SD−1, thisdistribution cannot be expressed in a closed form. In-stead, we utilize the MH framework to sample µk. Itis well known in the literature that the closer the pro-posal distribution is to the target posterior distribu-tion, the faster the convergence. We therefore use thefollowing proposal as an approximation to Eqn. (12):

q(µk|x, z,Σk) = p(µk)N (Log〈x〉k(µk); 0, ΣkNk

) (13)

See the supplemental material for more details. Theproposed mean µk is accepted according to the MHalgorithm, with probability Pr (accept) = min (1, r),where the Hastings ratio, r, is

r = p(x|z,µk,Σk)p(µk)q(µk|x,z,Σk)p(x|z,µk,Σk)p(µk)q(µk|x,z,Σk) (14)

=N (Log〈x〉k

(µk);0,Σk/Nk)

N (Log〈x〉k(µk);0,Σk/Nk)

∏i∈Ik

N (Logµk(xi);0,Σk)

N (Logµk(xi);0,Σk) .

We have used the fact that the distribution of means,p(µk), is uniform over the sphere.

Finally, given means, µ, and covariances, Σ, we samplenew labels, z, for all data, x, as

zi∝∼∑K

k=1πkN (Logµk(xi); 0,Σk)1[zi=k] , (15)

where∝∼ denotes sampling from the distribution pro-

portional to the right side, and the indicator function1[zi=k] is 1 if zi = k and 0 otherwise.

4.2 Sub-Cluster Split/Merge Proposals

We now describe the MH split-and-merge proposalsthat are specialized to the geometry of SD−1. Thepreviously-defined posterior distributions for direc-tional data uniquely define posterior inference in thesub-clusters [6]. When constructing split-and-mergemoves, joint proposals over the entire latent space,{z,µ,Σ}, must be constructed. The proposed labels,z, will be constructed from the inferred sub-clusters.Ideally, the parameters, µ and Σ, will be proposedfrom the true posteriors. However, as discussed pre-viously, no conjugate prior exists for µ. Hence wepropose the parameters from

µa ∼ q(µa|x, z) = N (Log〈x〉a(µa); 0,Σ?a) , (16)

Σa ∼ p(Σa|x, z, µa) , (17)

where Σ?a , arg maxΣ IW(∆+S〈x〉a , ν+Na). The scat-ter matrix, S〈x〉a , in T〈x〉aSD−1 is computed accordingto Eq. (11), and p(Σa|x, z, µa) is the true posteriordenoted in Eq. (9). We note that Eqn. (16) uses thecovariance Σ?a instead of Σa because Σa depends on µathrough the point of tangency.

The DP Sub-Cluster algorithm then deterministicallyconstructs moves from the sub-clusters. We extendthe algorithm to the DP-TGMM by splitting cluster ainto clusters b and c with:

zIa = split-a(z), (µb, µc) = (µa`, µar),

(Σb, Σc) ∼ p(Σb|x, z, µb)p(Σc|x, z, µc) , (18)

and by merging clusters b and c into cluster a with:

zIb∪Ic = merge-bc(z), µa ∼ q(µa|x, z),

Σa ∼ p(Σa|x, z, µa) , (19)

where split-a(z) splits cluster a into clusters b and cdeterministically based on the sub-cluster labels, andmerge-bc(z) merges clusters b and c into cluster a. Weshow in the supplement that this choice of proposalsresults in the following Hastings ratio for a split:

rsplit = αΓ(Nb)Γ(Nc)Γ(Na)

p(x|z,µ)p(µb)p(x|z,µ) q(µa|x, z) . (20)

As shown in [6], any proposed deterministic mergemove will be rejected with very high probability. Assuch, the DP Sub-Cluster incorporates a set of ran-domized split/merge proposals that are generated froma data-independent, two-dimensional Dirichlet distri-bution. We use this formulation as well, and introducethe following randomized split/merge proposals. Therandom splits are proposed as:

zIa ∼ DirMult(α/2, α/2),

(µb, µc) ∼ q(µb|x, z)q(µc|x, z),

(Σb, Σc) ∼ p(Σb|x, z, µb)p(Σc|x, z, µc) , (21)

A Dirichlet Process Mixture Model for Spherical Data

DP-GMM DP-TGMM

isotropic

anisotropic

Figure 3: Mean and standard deviation over ten sampler runs of normalized mutual information (NMI) andcluster-count for synthetic datasets of 30 isotropic (top) and anisotropic (bottom) clusters on S2. The colors forthe different algorithms are consistent across all plots. In the sphere-plots to the right it can be observed that,in contrast to the DP-TGMM, the DP-GMM fails to separate (top) or incorrectly splits (bottom) clusters.

and the random merges according to:

zIb∪Ic = merge-bc(z), µa ∼ q(µa|x, z),

Σa ∼ p(Σa|x, z, µa) . (22)

We show in the supplementary material that these re-sult in the following Hastings ratios:

rrandsplit = αΓ(α/2)2Γ(α+Na)Γ(Nb)Γ(Nc)

Γ(α)Γ(Na)Γ(α/2+Nb)Γ(α/2+Nc)

· p(x|z,µ)p(µb)p(x|z,µ)

q(µa|x,z)q(µb|x,z)q(µc|x,z) (23)

rrandmerge = Γ(α)Γ(Na)Γ(α/2+Nb)Γ(α/2+Nc)

αΓ(α/2)2Γ(α+Na)Γ(Nb)Γ(Nc)

· p(x|z,µ)p(x|z,µ)p(µb)

q(µb|x,z)q(µc|x,z)q(µa|x,z) . (24)

Note that while 〈x〉b, 〈x〉c, Sµb , and Sµc must be re-computed for the random split proposal, we efficientlyapproximate these quantities for the deterministic splitand the random merge from the statistics of clusters band c as described in the supplemental.

5 Experimental Results

In the following we compare the DP-TGMM infer-ence algorithm on synthetic data with ground-truthlabels against four related algorithms. Subsequently,we evaluate the clustering on real data, namely, surfacenormals extracted from Kinect depth images, and 20-dimensional semantic word vectors [32]. All MCMCinference algorithms are evaluated based on one sam-ple from the Markov chain after burn-in.

5.1 Comparisons on Synthetic Data

We generate ground-truth data on the 3D unit sphereby sampling from a 30-component mixture model withequi-probable classes. The cluster means are drawn

from a uniform distribution on the unit sphere andcovariances from an IW prior. As depicted to theright in Fig. 3 the datasets for evaluation encompassan isotropic as well as an anisotropic dataset.

We compare the DP-TGMM with two commonly-used optimization-based clustering algorithms, k-means [21] and spherical k-means (spkm) [10], as wellas with the finite symmetric Dirichlet approximation(FSD-TGMM) [25] to the DP-TGMM. Additionally,we show the performance of the DP-GMM, a BNP in-finite GMM, that does not exploit the geometry of thesphere. DP-GMM inference uses the sub-cluster-splitalgorithm [6]. All algorithms are initialized with a ran-dom labeling of the data. We use normalized mutualinformation (NMI) [45] between the groundtruth andthe inferred labels as a measure for clustering qualitywhich penalizes the use of superfluous clusters. Addi-tionally, we show the number of clusters per iteration,which changes only for the BNP models.

From the middle plots in Fig. 3 it can be seen thatthe inference for the DP-TGMM finds the true num-ber of clusters in both cases, while the DP-GMM doesnot. The FSD-TGMM method gives an incorrect es-timate of the number of clusters, which is consistentwith what was observed in [7]. This motivates the needfor our proposed sub-cluster inference algorithm. Thedepiction of the clustering results on the sphere to theright of Fig. 3, shows that the DP-GMM fails to sep-arate isotropic clusters and splits anisotropic clustersincorrectly. The parametric algorithms, k-means andspkm, were set to the true number of clusters, whichis unknown in many problems of interest.

The evolution of the NMI with iterations, depicted inthe left plots of Fig. 3, shows that the optimization-based methods quickly converge to a (sub-optimal)

Julian Straub, Jason Chang, Oren Freifeld, John W. Fisher III

RG

Bsp

km

DP

-GM

MD

P-T

GM

M

Figure 4: In the top row we show RGB images of the scene (only) for reference. We visualize the segmentationimplied solely by the clustering of normals using the spkm algorithm with k = 7 (2nd row), using the DP-GMM(3rd row) and the proposed DP-TGMM (bottom row). Note how spkm and DP-GMM fail to properly segmentthe scenes. The colors encode the association to inferred clusters and black denotes missing depth data.

solution. The sampling based algorithms generallyachieve better solutions. The DP-TGMM finds thebest fit to the data as it explicitly allows for anisotropicdistributions and respects the geometry of the sphere.

5.2 Surface Normals in Point-cloud Data

Surface normals extracted from point-clouds exhibitclusters on the unit sphere since planes in a scene cre-ate sets of normals pointing into the same direction.Hence, clustering these normals amounts to segment-ing the scene into planes with similar orientation. Weextract surface normals from raw Kinect depth imagesof the NYU V2 dataset [33] using the algorithm de-scribed in [24] and apply total variation regularization[40] to smooth the initial normal estimate.

We show a set of examples scenes in Fig. 4 and thesegmentation into planes with equal orientation as im-plied by the clustering of normals on the unit sphereobtained using three different algorithms. In the sec-ond row we display results from clustering with spkmwhere k = 7, in the third clusterings obtained usingthe DP-GMM sub-cluster algorithm and in the lastrow the segmentation obtained with the DP-TGMMinference algorithm. Note that the scene images in thefirst row are only for reference – only surface normalswere used as input to the algorithms. The DP-GMMas well as the DP-TGMM inference was initialized totwo clusters with the hyper-parameters of the IW prior

set to ν = 10k and ∆ = (12◦)2νI3×3. Each scene con-tains around 300k data points on S2.

The differences in segmentation illuminate the short-comings of spkm and DP-GMM. The spkm algorithmfinds a decent segmentation, but we get spurious clus-ters since the number of clusters is generally unknown.The inferred DP-GMM tends to under-segment thedata because it ignores the manifold of the data andhence does not properly split clusters of normals inthe presence of significant noise in the real data. Forexample in column two and five of Fig. 4 the floor andthe wall are not separated into distinct clusters. By re-specting the manifold as well as adopting a BNP modelthe DP-TGMM infers the intuitively correct segmen-tation as can be seen in the last row of Fig. 4.

5.3 Clustering of Semantic Word-Vectors

We extract 20-dimensional semantic word vectors [32]from the English Wikipedia corpus and filter out allwords with less than 100 counts to arrive at a set of41k semantic word vectors for English words. Notethat we normalize the word vectors to unit length be-fore clustering. This is motivated by the fact that [32]utilizes the cosine similarity to find the semanticallyclosest word to a given location in the vector space.The use of the cosine similarity is equivalent to theassumption that all the information about semanticproximity resides in the angular difference.

A Dirichlet Process Mixture Model for Spherical Data

“finance” “music” “religion ” “leisure” “government” “food”funding symphonic orthodoxy malls parliamentary tomatoesprospective operatic orthodox hotels enacted edibleloans soloists evangelical dining parliament fruitfinancing orchestral christians nightlife delegation meatsfunds music primacy outdoor unanimously meatcontracts waltz preaching upscale granting vegetablescompensation trios doctrines shopping mandate juiceregulations lute rabbis restaurants constitutional bakedassets soloist clergy taverns citizenship corninvestors flute catholicism shops committee tasting

(a) Top: most likely words for 6 clusters. Bottom: log prob-ability and histogram over condition numbers for clusters.

“Europeans”

“non-Europeans”

“currencies”“countries”

(b) Dimensionality reduction in the tangent space byprojecting onto the eigenvectors of the two largesteigenvalues of the cluster’s covariance matrix.

Figure 5: Evaluation of DP-TGMM inference on 20D semantic word vectors trained on the Wikipedia corpus.

Therefore, by clustering the semantic word-vectors bytheir directions we obtain clusters of semantically sim-ilar words as can be observed in the table of Fig. 5a.The table lists the ten most likely words of a subsetof the clusters obtained when running the DP-TGMMinference algorithm. We start the algorithm from 20centroids and run it for 800 iterations. After about 250iterations the algorithm converges to 96 clusters. Notethat this clustering is different from conventional topicmodeling, which relies on document-level word counts.Semantic word-vectors depend on nearby words, andour clustering disregards document groupings.

To validate our hypothesis that real directional dataexhibits anisotropic distributions on the sphere, wecompute the condition number of the inferred clus-

ter covariance matrices κ(Σk) = max[σ(ΣK)]min[σ(ΣK)] where

σ(ΣK) is the set of all eigenvalues of Σk. The con-dition number thus serves as a measure for how ellip-tical the clusters are: κ(Σk) ≈ 1 means the clusteris isotropic whereas κ(Σk) � 1 indicates an ellipticalor anisotropic distribution. The spread-out histogramover condition numbers shown in Fig. 5a indicates thatthe inferred covariances are indeed anisotropic.

To demonstrate the analysis our model affords, weshow in Fig. 5b the conceptual distribution of wordsin an inferred cluster when projected onto the 2D co-ordinate system defined by the two largest eigenval-ues of the cluster’s covariance matrix in the tangentspace. We show 20 words that are at the extremes ofboth axes. In one direction the meaning of the wordschanges from non-European to European countries.On the other coordinate axes we find a progressionfrom a mix of different countries to their currencies.We note that such analysis is made possible by model-

ing the data as a mixture of anisotropic distributions.

6 Conclusion

In this work we introduce the DP-TGMM, a DP mix-ture model over Gaussian distributions in multipletangent spaces to the unit sphere in RD. Aimed atmodeling directional data, this Bayesian nonparamet-ric model does not only adapt to the complexity ofthe data but also describes anisotropic distributionson the sphere. Experiments on synthetic data demon-strate that the proposed model is more expressivein describing directional data than other commonly-used models. Moreover, we have shown the scala-bility and effectiveness of the inference algorithm aswell as the applicability and versatility of the modelon batches of 300k real-world 3D surface normalsand on 20-dimensional semantic word-vectors for 41kEnglish words. All inference code can be found athttp://people.csail.mit.edu/jstraub/.

Future work should investigate the extension of thismodel to other Riemannian manifolds. Anotherpromising research direction is embedding DP-TGMMin a hierarchical structure (akin to the HierarchicalDP-GMM for RD-valued data) to allow informationsharing between batches of data in applications suchas protein backbone configuration modeling.

Acknowledgements

We thank Karthik Narasimhan and Ardavan Saeedifor pointing us to the work on semantic word vec-tors. This work was partially supported by the ONRMURI N00014-11-1-0688, ARO MURI W911NF-11-1-0391, and the AFRL MIMFA program.

Julian Straub, Jason Chang, Oren Freifeld, John W. Fisher III

References

[1] A. Banerjee, I. S. Dhillon, J. Ghosh, S. Sra, andG. Ridgeway. Clustering on the unit hypersphere us-ing von Mises-Fisher distributions. JMLR, 2005.

[2] M. Bangert, P. Hennig, and U. Oelfke. Using an infi-nite von Mises-Fisher mixture model to cluster treat-ment beam directions in external radiation therapy.ICMLA, 2010.

[3] C. Bingham. An antipodally symmetric distributionon the sphere. The Annals of Statistics, 1974.

[4] C. M. Bishop et al. Pattern recognition and machinelearning, volume 1. springer New York, 2006.

[5] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent Dirich-let allocation. JMLR, 2003.

[6] J. Chang and J. W. Fisher III. Parallel sampling of dpmixture models using sub-clusters splits. NIPS, 2013.

[7] J. Chang and J. W. Fisher III. MCMC sampling inHDPs using sub-clusters. NIPS, 2014.

[8] S. B. Choe. Statistical analysis of orientation trajec-tories via quaternions with applications to human mo-tion. PhD thesis, The University of Michigan, 2006.

[9] D. B. Dahl, Z. Bohannan, Q. Mo, M. Vannucci, andJ. Tsai. Assessing side-chain perturbations of the pro-tein backbone: A knowledge-based classification ofresidue ramachandran space. JMB, 2008.

[10] I. S. Dhillon and D. S. Modha. Concept decomposi-tions for large sparse text data using clustering. Ma-chine learning, 2001.

[11] M. P. do Carmo. Riemannian Geometry. BirkhauserVerlag, Boston, MA, 1992.

[12] W. Feiten, M. Lang, and S. Hirche. Rigid motion es-timation using mixtures of projected gaussians. FU-SION, 2013.

[13] T. Ferguson. A Bayesian analysis of some nonpara-metric problems. The annals of statistics, 1973.

[14] N. I. Fisher. Statistical Analysis of Circular Data.Cambridge University Press, 1995.

[15] R. Fisher. Dispersion on a sphere. Proc. R. Soc. A:Math. & Phy. Sci., 1953.

[16] O. Freifeld, S. Hauberg, and M. J. Black. Model trans-port: Towards scalable transfer learning on manifolds.CVPR, 2014.

[17] Y. Furukawa, B. Curless, S. M. Seitz, and R. Szeliski.Reconstructing building interiors from images. ICCV,2009.

[18] A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson,A. Vehtari, and D. B. Rubin. Bayesian data analysis.CRC press, 2013.

[19] J. S. Goddard and M. A. Abidi. Pose and motion esti-mation using dual quaternion-based extended kalmanfiltering. Photonics West’98 Electronic Imaging, 1998.

[20] K. Grove and H. Karcher. How to conjugatec 1-closegroup actions. Mathematische Zeitschrift, 1973.

[21] J. A. Hartigan and M. A. Wong. Algorithm as 136: Ak-means clustering algorithm. Applied statistics, 1979.

[22] W. K. Hastings. Monte Carlo sampling methods us-ing Markov chains and their applications. Biometrika,1970.

[23] S. Hauberg, F. Lauze, and K. Pedersen. UnscentedKalman filtering on Riemannian manifolds. JMIV,2012.

[24] D. Holz, S. Holzer, and R. B. Rusu. Real-Time PlaneSegmentation using RGB-D Cameras. Proc. of theRoboCup Symposium, 2011.

[25] H. Ishwaran and M. Zarepour. Exact and approx-imate sum representations for the Dirichlet process.

Canadian Journal of Statistics, 2002.[26] H. Karcher. Riemannian center of mass and mollifier

smoothing. Commun. Pure and Appl. Math., 1977.[27] H. Karcher. Riemannian center of mass and so called

karcher mean. arXiv:1407.2087, 2014.[28] J. T. Kent. The Fisher-Bingham distribution on the

sphere. Journal of the Royal Statistical Society, 1982.[29] K. P. Lennox, D. B. Dahl, M. Vannucci, and J. W.

Tsai. Density estimation for protein conformationangles using a bivariate von Mises distribution andBayesian nonparametrics. JASA, 2009.

[30] S. N. MacEachern. Estimating normal means witha conjugate style Dirichlet process prior. Commun.Stat. - Simul. and Comp., 1994.

[31] K. V. Mardia and P. E. Jupp. Directional statistics,volume 494. John Wiley & Sons, 2009.

[32] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, andJ. Dean. Distributed representations of words andphrases and their compositionality. NIPS, 2013.

[33] P. K. Nathan Silberman, Derek Hoiem and R. Fer-gus. Indoor segmentation and support inference fromRGBD images. ECCV, 2012.

[34] D. Peel, W. J. Whiten, and G. J. McLachlan. Fit-ting mixtures of kent distributions to aid in joint setidentification. JASA, 2001.

[35] X. Pennec. Probabilities and statistics on Riemannianmanifolds: Basic tools for geometric measurements.NSIP, 1999.

[36] J. Pitman. Combinatorial stochastic processes. Tech-nical Report 621, U.C. Berkeley Department of Statis-tics, 2002.

[37] C. Ramakrishnan and G. Ramachandran. Stereo-chemical criteria for polypeptide and protein chainconformations: Ii. allowed conformations for a pairof peptide units. Biophysical Journal, 1965.

[38] J. Reisinger, A. Waters, B. Silverthorn, and R. J.Mooney. Spherical topic models. ICML, 2010.

[39] C. P. Robert and G. Casella. Monte Carlo statisticalmethods. Springer, 1999.

[40] G. Rosman, Y. Wang, X.-C. Tai, R. Kimmel, andA. M. Bruckstein. Fast regularization of matrix-valuedimages. ECCV, 2012.

[41] J. Sethuraman. A constructive definition of Dirichletpriors. Technical report, DTIC Document, 1991.

[42] E. Simo-Serra, C. Torras, and F. Moreno-Noguer.Geodesic finite mixture models. BMVC, 2014.

[43] S. Sommer, F. Lauze, S. Hauberg, and M. Nielsen.Manifold valued statistics, exact principal geodesicanalysis and the effect of linear approximation.ECCV, 2010.

[44] J. Straub, G. Rosman, O. Freifeld, J. J. Leonard, andJ. W. Fisher III. A mixture of Manhattan frames:Beyond the Manhattan world. CVPR, 2014.

[45] A. Strehl and J. Ghosh. Cluster ensembles—a knowl-edge reuse framework for combining multiple parti-tions. JMLR, 2003.

[46] A. Strehl, J. Ghosh, and R. Mooney. Impact of simi-larity measures on web-page clustering. AAAI, 2000.

[47] D. Ting, G. Wang, M. Shapovalov, R. Mitra, M. I. Jor-dan, and R. L. Dunbrack Jr. Neighbor-dependent ra-machandran probability distributions of amino acidsdeveloped from a hierarchical Dirichlet process model.PLoS computational biology, 2010.

[48] Wikipedia. Database download, June 2014.[49] S. Zhong. Efficient online spherical k-means cluster-

ing. IJCNN, 2005.