Finding New Malicious Domains Using VariationalBayes on Large-Scale Computer Network Data

Vojtech Letal, Tomas Pevny ⇤

Czech Technical University in PragueCzech Republic

{letal.vojtech,pevnak}@gmail.com

Vaclav Smıdl, Petr Somol⇤UTIA, Czech Academy of Sciences

Czech Republic{smidl,somol}@utia.cas.cz

Abstract

The common limitation in computer network security is the reactive nature ofdefenses. A new type of infection typically needs to be first observed live, be-fore defensive measures can be taken. To improve the pro-active measures, wehave developed a method utilizing WHOIS database (database of entities that hasregistered a particular domain) to model relations between domains even thosenot yet used. The model estimates the probability of a domain name being usedfor malicious purposes from observed connections to other related domains. Theparameters of the model is inferred by a Variational Bayes method, and its effec-tiveness is demonstrated on a large-scale network data with millions of domainsand trillions of connections to them.

1 Motivation

In network security it is increasingly more difficult to react to influx of new threats. Reactive defensesolutions based on constant anti-virus signature updates do not scale. We aim at revealing a signifi-cant fraction of zero-day (previously unknown) network threats pro-actively by inferring in advancethe probability of domain names (domains) to be used for malicious purposes to which connectionshave not been yet observed. The idea is to use external information to link domains to those withalready observed and investigated connections. The proposed model builds upon Bayesian statisticswith unknowns inferred by variational methods. The model is applicable to estimate probabilityof being involved in malicious activities of any computer network entity, given a suitable exter-nal database is available to connect them. The concept is demonstrated on estimating fraction ofmalware-related connections to millions of domains related through records stored in publicly ac-cessible WHOIS databases. The experimental results show the method is robust enough to be usedin realistic large-scale computer network setting with significant amount of missing data.

1.1 Notation

Let x 2 0, 1 denote a realization of a random variable, which in this work denotes maliciousnessof a network connection (e.g. request of a web browser) to a domain d (e.g. example.com). Aconnection has value zero if it is benign and one if it is considered to be malicious by some intrusiondetection system (IDS). The set of all observed connections used in the inference is denoted byX , and the set of all domains D. Since x is a realization of a random variable, X contains manyconnections to the same domain d denoted X (d). A mapping d(x) returning a domain of a requestx will be useful below.

Let md

be a parameter of the Bernoulli distribution of domain d determining the probability withwhich requests to d are blocked. M is a set of m

d

of all domains, and those are the parameters we⇤Tomas Pevny and Petr Somol are also with Cisco Systems Inc.

1

want to infer. Individual domains d are linked together by informations stored in WHOIS database,to which it is referred to as keys. L is the set of all keys in the WHOIS database. The role of keys inthe model is symmetric, it is therefore assumed L = K

o

[Ke

[Kn

[Kp

, where Kn

,Ko

,Ke

,Kp

aresets of all registrants names, organizations, e-mail addresses and postal addresses. Note that if twokeys from different categories share the same value, they are considered to be distinct, e.g. registrantwith the name Foo and organization with the name Foo are considered to be two different keys inL. The particular key (composed from values in the WHOIS database) of domain d is a quadruplek(d) = (k

n

(d), k

o

(d), k

e

(d), k

p

(d)) of indexes from L, corresponding to k

n

(d) 2 Kn

, k

o

(d) 2 Ko

,

k

e

(d) 2 Ke

, and k

p

(d) 2 Kp

.

2 The model

The goal of the proposed system is to estimate probability, md

, that a web request to domain d

(e.g. example.com) is related to a malicious activity (such requests are further called malicious).The system estimates these probabilities from frequencies with which individual requests are clas-sified to be malicious by some Intrusion Prevention System (IPS) and uses informations about do-mains provided by the WHOIS database to link domains together. By virtue of this link, the systemprovides an estimate for domains to which few or none requests have been observed and / or in-spected by the IDS. We assume that the WHOIS database provides for every domain a quadruplek(d) = (k

n(d), ko(d), ke(d), kp(d)), where k

n(d) is the name of the registrant of domain d, ko(d) its

organization, ke(d) its contact email address, and k

p(d) the postal address. We model the aboveas follows. The probability of a request x directed to domain d being malicious follows Bernoullidistribution

p

b

(x|md

) = m

x

d

(1�m

d

)

1�x

, (1)where it is assumed that all requests are independent. The parameter m

d

, which is equal to thefraction of blocked request to d, is the parameter we would like to infer based on observed data Xand the WHOIS record K. To do so, we introduce a Beta prior on the value of m

d

in form

p

d

=

m

a

k(d)�1d

(1�m

d

)

b

k(d)�1

�(a

k(d), bk(d)), (2)

where parameters ak(d) and b

k(d) are parameters of the Beta-prior depending on the key k(d), whichis an unique quadruple retrieved from the WHOIS database. Formulation (2) can be possibly usedto infer all m

d

2 M using maximum likelihood approach, but it makes Beta priors a

k(d), bk(d)

unique to each quadruple k(d), which means that two records differing in one or more sub-keys(e.g. postal address) have to be treated independently. Moreover both priors would need to beguessed sufficiently well, which would be very difficult. To resolve both problems, priors a

k(d) andb

k(d) are factorised such that each factor depends only on the portion of the key (e.g. e-mail address)and a Gamma prior is put on each factor [4]. Specifically,

a

k(d) = a

k

n

(d)ako

(d)ake

(d)akp

(d) and b

k(d) = b

k

n

(d)bko

(d)bke

(d)bkp

(d), (3)

where k

n

(d) 2 Kn

, k

o

(d) 2 Ko

, k

e

(d) 2 Ke

, k

p

(d) 2 Kp

with Kn

, Ko

, Ke

, and Kp

being setof all unique names, organizations, e-mail addresses, and postal addresses respectively. To simplifynotation, we introduce L = K

o

[Ke

[Kp

[Kn

(if a same value of key appears in for example Ko

and Ke

, they are treated as two different keys). With this notation we index a

k

n

(d), ako

(d), ake

(d),

and a

k

p

(d) as al

, l 2 L, and we can write a

k(d) =Q

l2k(d) al and b

k(d) =Q

l2k(d) bl. Finally, weintroduce Gamma prior on a

l

and b

l

, l 2 L yielding to he final model

p(a

l

) =

v

a

u

a

�(u

a

)

a

u

a

�1l

e

�v

a

a

l and p(b

l

) =

v

b

u

b

�(u

b

)

b

u

b

�1l

e

�v

b

b

l

. (4)

The final joint-probability function of the model is

p(M,A,B|X , ✓) =

Y

x2Xm

x

d(x)(1�m

d(x))1�x

Y

d2D

m

a

k(d)�1d

(1�m

d

)

b

k(d)�1

�

�a

k(d), bk(d)

� ·

Y

l2L

v

a

u

a

�(u

a

)

a

u

a

�1l

e

�v

a

a

l

v

b

u

b

�(u

b

)

b

u

b

�1l

e

�v

b

b

l

,

(5)

where a

k(d) and b

k(d) are from (3).

2

Algorithm 1 Variational Bayes inference of model parameters M = [d2Dmd

, A = [l2Lal, B =

[l2Lbl. The convergence criterion is the change of expected value of m

d

.

1. choose the parameters ⇥ = (u

a

, v

a

, u

b

, v

b

)

2. choose initial estimates of q(al

) and q(b

l

)

3. For it = 1 to max iterations(a) 8d 2 D recompute q(m

d

)

8d 2 D evaluate logm

d

V

, log(1�m

d

)

V

(b) 8al

2 A recompute q(a

l

)

8al

2 A evaluate a

l

V

, log al

V

(c) 8bl

2 B recompute q(b

l

)

8bl

2 B evaluate b

l

V

, log bl

V

(d) If convergence criteria < threshold Then break

3 Inference of parameters by Variational Bayes

Since an analytical solution of (5) does not exist due to the integral of the beta distribution not hav-ing a closed-form solution, a Variational Bayes method [5, 1] approximating posterior distributionimplied by (5) using a product of marginal distributions is evaluated

p(M,A,B|X , ✓) ⇡ q(M,A,B) =Y

d2Dq(m

d

)

Y

l2Lq(a

l

)q(b

l

). (6)

The Variational Bayes method minimizes the KL-divergence between the approximation (6) and thejoint pdf (5) by iteratively optimizing individual marginals q(m

d

), q(a

l

), and q(b

l

) while keepingall other marginals fixed. After sufficient number of iterations the algorithm converges to a localminimum. The algorithm is outlined in Algorithm 1 with the marginals1 being

q(m

d

) ⇠ Beta

0

@Y

l2k(d)

a

l

V

+

X

x2X (d)

x,

Y

l2k(d)

b

k

V

+

X

x2X (d)

(1� x)

1

A, (7)

q(a

l

) ⇠ Gamma

0

@u

a

+

X

{d2D|l2k(d)}

⇣

d,k(d), va �X

{d2D|l2k(d)}

a

k(d)\lV

logm

d

V

1

A (8)

q(b

l

) ⇠ Gamma

0

@u

b

+

X

{d2D|l2k(d)}

⇣

d,k(d), vb �X

{d2D|l2k(d)}

b

k(d)\lV

logm

d

V

1

A, (9)

(10)where

⇣

d,k(d) =

h (a

k(d)

V

+ b

k(d)

V

)� (a

k(d)

V

) + b

k(d)

V

0(a

k(d)

V

+ b

k(d)

V

)(log b

k(d)

V

� log b

k(d)

V

)

i, (11)

a

k(d)\l =

Ql

02k(d)^l

0 6=l

a

l

0 , bk(d)\l =

Ql

02k(d)^l

0 6=l

b

l

0, and the ˆ· denotes expected value of the

variable. Exact variational marginals do not have an analytical solutions due to the intractability ofthe Beta distribution. Therefore we have used approximation proposed in [4] to obtain q(a

l

) andq(b

l

). The approximations are based on Taylor expansion of the logarithm of the Beta distributionin a

k(d)

V

, bk(d)

V

from previous iteration. Since the Taylor expansion is valid only for ak(d)

V

, b

k(d)

V

> 1,both variables are set to one if they are smaller.

4 Experimental results

The proposed method was evaluated on data obtained from web-logs Cisco’s Cloud Web Secu-rity [3], which is a cloud web proxy scrutinizing the traffic for the presence of known malware and

1Derivation of the marginals can be found in the supplemental material.

3

10

�410

�310

�20

0.2

0.4

0.6

0.8

1

FPR

November – January

PTPIV

10

�410

�310

�20

0.2

0.4

0.6

0.8

1

FPR

October – November

PTPIV

10

�410

�310

�20

0.2

0.4

0.6

0.8

1

FPR

TPR

September – October

PTPIV

Figure 1: Comparison of the proposed method (IV) with the Probabilistic Threat Propagation (PTP).

other threats. For the experiments we have used traffic from the first week of September, October,November 2014, and January and Fabruary 2015. Each week of traffic contained approximately7.5 · 109 connections (size of the set X ) out of which approximately 20 · 103 were deemed as ma-licious) to approximately 2 · 106 domains (size of the set D) with approximately 10

5 keys retrievedfrom the WHOIS database.

The data allows a natural division into training and testing data, where we have used the data from aprevious month for training (inferring values in sets A = [

l2Lal, B = [l2Lbl, and M = [

d2Dmd

)and the data from the next month for the evaluation.

The proposed method was compared to Probabilistic Threat Propagation [2], which is a methodthat propagates a probability of certain network node being malicious based on connection graph.The method extrapolates maliciousness of “tips”, which are domains used for malicious purposes,to other connected domains, which were domains sharing at least one key in WHOIS database. Inour experiments “tips” were domains with at least 20% blocked connection, which was a fractiondetermined to avoid trivial false positives like yahoo.com.

Figure 1 shows ROC curves when the inference was done on data captured in September, October,and November, and evaluated on October, November, and January respectively. The ROC curve wasobtained by changing the threshold on m

d

from which the domain would be considered maliciousand counting correctly classified flows (blocked vs. not blocked). The false positive rate is drawn inlogarithmic scale, because in security applications only very low false positive rate are interesting.Therefore only the region from zero to one percent false positive rate is shown to decide whichalgorithm is better. We observe that ROC curves of both methods intersect, but that of the proposedmethod is generally above (better) in the region of interest. Moreover, the proposed method does notrequire known “tips”, which is an important feature for practical deployment, as it can be executedautonomously.

5 Conclusion

We have defined and verified a Bayesian model to infer probability of a network entity being in-volved in malicious activities. The important feature for practice is that the model propagates prob-ability from entities with observed connections to those without using external information relatingentities together. The model was instantiated to enable preventive blacklisting of yet unobserveddomains using information about observed HTTP request blocks and domain registration records inthe WHOIS database. The scalability of the model was shown on modeling millions of domainsusing trillions of web requests.

6 Acknowledgement

The work of V. Letal, P. Somol and T. Pevny has been partially supported by Czech Science Foun-dation project 15-08916S.

4

References

[1] Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and

Statistics). Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2006.[2] Kevin M Carter, Nwokedi Idika, and William W Streilein. Probabilistic threat propagation for

malicious activity detection. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE

International Conference on, pages 2940–2944. IEEE, 2013.[3] Inc. Cisco Systems˙ Cloud web security, 2015. ¡http://www.cisco.com/c/en/us/

products/security/cloud-web-security/index.html¿.[4] Zhanyu Ma and Arne Leijon. Bayesian estimation of beta mixture models with variational

inference. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 33(11):2160–2173, 2011.

[5] Vaclav Smıdl and Anthony Quinn. The variational Bayes method in signal processing. SpringerScience & Business Media, 2006.

5

Suplemental material for Finding New Malicious

Domains Using Variational Bayes on Large-Scale

Computer Network Data

Vojt

ˇ

ech L

´

etal, Tom

´

a

ˇ

s Pevn´y

⇤

Department of Computer ScienceCzech Technical University in Prague

Czech Republic{letal.vojtech,pevnak}@gmail.com

V

´

aclav

ˇ

Sm´ıdl, Petr Somol

⇤

Institute of Information Theory and AutomationCzech Academy of Sciences

Czech Republic{smidl,somol}@utia.cas.cz

1 Recapitulation of variational Bayes to introduce notation

The recapitulation of Variational Bayes follows [1] page 464. To simplify the notation, we assumep(z, d) to be the joint distribution of data d and parameters z we are interesting in. Furthermore, p(x)is the marginal distribution of the data and q(z) is a distribution by which we want to approximatep(z|x) (distribution of parameters given data). Naturally it is assumed z to be multi-dimensionalwith k components. It holds that

log p(x) =

Z

Zq(z) log p(x)dz

=

(1)

Z

Zq(z) log

✓p(x, z)

p(z|x)q(z)

q(z)

◆dz

=

Z

Zq(z) log

p(x, z)

q(z)

dz �Z

Zq(z) log

p(z|x)q(z)

dz

=

(2)E(q(z)) + DKL (q(z)||p(z|x)) ,

(1)

where in (1) we have used the fact that p(x, z) = p(z|x)p(x) and in (2) we have recognized KL-divergence in the second term and denoted the first one by E(q(z)) =

RZ q(z) log

p(x,z)q(z) dz.

The KL-divergence DKL (q(z)||p(z|x)) measures the closeness of q(z) and p(z|x). While it is nota true distance, it is non-negative and equal to zero iff q(z) = p(z|x). Since the KL-divergence isalways positive, by maximizing E(q(z)) we minimize the DKL (q(z)||p(x|z)) for which we usuallydo not have a closed form solution. Thus finding the best approximation q(z) amounts to maximizeE(q(z)).

⇤Tomas Pevny and Petr Somol are also with Cisco Systems Inc.

1

Factorized models assume that q(z) is a product of independent probability distributions, i.e. q(z) =Qi

q

i

(z

i

). With this assumption one can readily write

E(q(z)) =Z

Zq(z) log

p(x, z)

q(z)

dz

=

Z

Zj

q

j

(z

j

)

0

@Z

Z\j

Y

i 6=j

q

i

(z

i

) log p(x, z)dz\j

1

Adz

j

�X

i

Z

Zq(z) log q

i

(z

i

)dz,

(2)

where the notation \j means with respect or over to all variables except j. Let now introduce

log p

j

(x, z) =

Z

Z\j

Y

i 6=j

q

i

(z

i

) log p(x, z)dz\j = EZ\j⇠q\j [log p(x, z)] (3)

and denote ⌘

i

=

RZi

q

i

(z

i

) log q

i

(z

i

)dz

i

. The the above can be simplified to

E(q(z)) =Z

Zj

q

j

(z

j

) log

p

j

(x, z)

q

j

(z

j

)

dz

j

�X

i 6=j

⌘

j

=�DKL (qj(zj)||pj(x, z))�X

i 6=j

⌘

j

(4)

Now imagine that we optimize E(q(z)) only with respect to q

j

(z

j

). The the sum on the right-handis independent and the KL term is maximized if the KL-divergence is equal to zero, which is whenq

j

(z

j

) = p

j

(x, z).

2 Extended factorized approximation method

The problem we face in our work is that the expectation EZ\j⇠q\j [log p(x, z)] is not tractable dueto the integration of the logarithm of the beta function. The extended factorized approximationproposes to find a lower bound on E(q(z)) and maximise this lower bound instead of E(q(z)). Bymaximising the lower bound iteratively we can still asymptotically reach the optimum. Specifically,Ref. [2] search a lower bound on p(x, z) (further denoted as p(x, z)) such that it holds that

Zq(z) log p(x, z)dz �

Zq(z) log p(x, z).dz (5)

The last inequality trivially holds iff p(x, z) � p(x, z).

The extended factorized approximation method maximizesRZ q(z) log

p(x,z)q(z) dz instead of E(q) by

using the variational Bayes method. In every iteration, the lower bound (or envelope) is updated, bywhich the same solution can be reached as if the original E(q) would be optimised.

Derivation of marginals

The joint pdf function is

p(M,A,B|X , ✓) =

Y

x2Xm

x

d(x)(1�m

d(x))1�x

Y

d2D

m

ak(d)�1d

(1�m

d

)

bk(d)�1

�

�a

k(d), bk(d)

� ·

Y

l2L

v

a

ua

�(u

a

)

a

ua�1l

e

�vaalv

b

ub

�(u

b

)

b

ub�1l

e

�vbbl,

(6)

2

The logarithm of it, which will prove useful immediately can be written as

log E(M,A,B|X , ✓) =

X

d2D

X

x2X (d)

x logm

d

+ (1� x) log(1�m

d

)

+

X

d2D(a

k(d) � 1) logm

d

+ (b

k(d) � 1) log(1�m

d

)� log �

�a

k(d), bk(d)

�

+

X

l2Lu

a

log v

a

+ (u

a

� 1) log a

l

� v

a

a

l

� log�(u

a

)

+

X

l2Lu

b

log v

b

+ (u

b

� 1) log b

l

� v

b

b

l

� log�(u

b

)

(7)

To use the Variational Bayes, we are interested in following quantities

log q(m

d

) =ED\d,A,B [log (E(M,A,B|X , ✓))] ,

log q(a

l

) =ED,A\al,B [log (E(M,A,B|X , ✓))] ,

log q(b

l

) =ED,A,B\bl [log (E(M,A,B|X , ✓))] .

(8)

Again the notation D\d denotes the set of domains D without the element d, and similarly for A andB. Note that for brevity, all terms not depending on the variable omitted from the expectation willbe skipped, as they will become part of the normalizing constant making q(·) a proper probabilitydistribution. We therefore use / instead of = to signalized the equality up to multiplicative constant.

Derivation of log q(m

d

)

For the log q(m

d

) holds

log q(m

d

) /X

x2X (d)

x logm

d

+ (1� x) log(1�m

d

)

+ (a

k(d) � 1) logm

d

+ (b

k(d) � 1) log(1�m

d

)

/ logm

d

0

@X

x2X (d)

x+ a

k(d) � 1

1

A+ log(1�m

d

)

0

@X

x2X (d)

(1� x) + b

k(d) � 1

1

A

(9)

in which we recognize a logarithm of the beta distribution. Therefore

q(m

d

) = Beta

0

@X

x2X (d)

x+ a

k(d)

V

,

X

x2X (d)

(1� x) + b

k(d)

V

1

A, (10)

whereˆ· denotes expected value of the variable.

Derivation of q(a

k

) and q(b

k

)

Since the derivation of q(ak

) and q(b

k

) follows exactly the same steps, it is shown only fro q(a

k

).For it holds

log q(a

l

) /EB,A\al

2

4X

{d2D|l2k(d)}

a

l

a

k(d)\l logmd

� log �(a

alak(d)\l , bk(d))

�(u

a

� 1) log a

l

� v

a

a

l

, ]

(11)

where a

k(d)\l =

Ql

02k(d)\l al0 and likewise for bk(d)\l. It is the term � log �(a

k(d), bk(d)) whichcauses the problem, as otherwise q(a

k

) would have the desired form of Gamma distribution, as theposterior would be the same as the prior. A remedy is to lower bound log q(a

l

) by deriving lowerbound of � log �(a

k(d), bk(d)), instead. By maximising the new lower bound we maximize the jointpdf as well. The derivation follows the steps [2] with the difference that here a

k(d) =Q

l2k(d) al. Itrelies on the two following properties copied from [2] without the proof.

3

Property 1. The normalization coefficient of Beta distribution can be approximated in point x0

using pseudo-Taylor [2] approximation as

� log �(x, y) � � log �(x0, y) + [ (x0 + y)� (x0)]x0(log x� log x0), (12)

where the is Digamma function, which is defined as first derivative of a logarithm of Gamma

function. This inequality holds for y > 1, and x, x0 2 R.

Property 2. Digamma function can be approximated in point y0 using pseudo-Taylor approximation

as

(x0 + y) � (x0 + y0) +

0(x0 + y0)y0(log y � log y0), (13)

where the

0is Trigamma function, which is defined as second derivative of a logarithm of Gamma

function. This inequality holds for y > 1, and x, x0 2 R.

The expectation of � log �(a

k(d), bk(d)) can be lower-bounded as

EB,A\al

⇥� log �(a

k(d), bk(d))⇤�(1)

EB,A\al

⇥� log �(a0, b

k(d))+

+

⇥� (a0 + b

k(d))� (a0)�a0

�log a

k(d) � log a0

�⇤⇤

/(2)

EB⇥a0

� (a0 + b

k(d))� (a0)�log a

l

⇤,

(14)

where in (1) we have used Property 1, and in (2) we have dropped terms not depending on a

l

andused the fact that log a

k(d) =P

l2k(d) log al.

To further break the term E

B

⇥ (a0 + b

k(d))⇤

we use the Property 2 as

EB⇥ (a0 + b

k(d))⇤�EB

⇥ (a0 + b0) +

0(a0 + b0)b0(log b

k(d) � log b0)⇤

= (a0 + b0) +

0(a0 + b0)b0

�EB[log b

k(d)]� log b0

�,

(15)

where the approximation again used Taylor expansion in point b0 and the expectation opera-tor is moved deeper inside the brackets. Note that no terms can be dropped in (15), becauseEB

⇥ (a0 + b

k(d))⇤

is multiplicative of log al

in (14).

By substituting (15) into (14) we obtain

EB,A\al

⇥� log �(a

k(d), bk(d))⇤�

⇥EB

⇥ (a0 + b

k(d))⇤� (a0)

⇤a0 log al

�⇥ (a0 + b0)� (a0) + b0

0(a0 + b0)(EB[log b

k(d)]� log b0)⇤a0 log al

= ⇣

d,al log al,

(16)

where ⇣d,al =

⇥ (a0 + b0)� (a0) + b0

0(a0 + b0)(EB[log b

k(d)]� log b0)⇤a0.

Finally, by substituting (16) into (18) a final lower bound on log q(a

l

) is obtained as

log q(a

l

) /EB,A\al

2

4a

k

X

{d2D|k2k(d)}

a

k(d)\k logmd

+ ⇣

d,al log al

�(u

a

� 1) log a

k

� v

a

a

k

] ,

(17)

where a Gamma distribution can be recognized. Hence

log q(a

l

) = Gamma

0

@u

a

+

X

{d2D|k2k(d)}

⇣

d,al , va �X

{d2D|k2k(d)}

a

k(d)\klogmd

V

1

A,

(18)

where againˆ· is the expected value.

In the algorithm described in the paper a0 = a

k(d)

V

and b0 = b

k(d)

V

from the previous iteration.

4

References

[1] Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and

Statistics). Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2006.[2] Zhanyu Ma and Arne Leijon. Bayesian estimation of beta mixture models with variational

inference. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 33(11):2160–2173, 2011.

5