Cooperative Swarms for Clustering Phoneme Data

8/3/2019 Cooperative Swarms for Clustering Phoneme Data

1/5

COOPERATIVE SWARMS FOR CLUSTERING PHONEME DATA

Abbas Ahmadi, Fakhri Karray, Mohamed Kamel

Pattern Analysis and Machine Intelligence Lab,

University of Waterloo, Canada

ABSTRACT

A new approach for clustering phoneme data is proposed. The

proposed approach mimics the behavior of biological swarms

seeking food located in different places. Best locations for

finding food are in dense areas and, in a same time, in regions

far from other places. We tackle phoneme clustering problem

using particle swarm optimization(PSO) and utilizing multi-

ple cooperating swarms to obtain cluster centers. The pro-

posed approach is evaluated on phoneme data of TIMIT database.

Experimental results show that the proposed clustering ap-

proach outperforms single swarm-based clustering as well as

k-means, k-harmonic means, and fuzzy c-means clustering

approaches.

Index Terms Cooperative systems, pattern clustering

methods, optimization methods, speech recognition.

1. INTRODUCTION

Using a streamlined clustering technique is a critical issue in

speech recognition. Speech recognition consists of two main

steps, phoneme recognition and word recognition. Phoneme

recognition is a crucial step in speech recognition, since it

forms the basis of mapping phonemes into words. Thus, it

plays an important role in constructing a powerful speech

recognition system. A phoneme is a smallest meaningful dis-

tinguishable unit in a languages phonology. Since the to-

tal number of phonemes for each language is finite, the goal

of phoneme recognition is to classify a speech signal into

phonemes with a given set of speech features [1]. To form

the architecture of a phoneme recognition system, a clustering

task is usually required; for instant, in modular approach for

speech recognition, clustering is applied to divide the problem

into subtasks [2]. Also, when applying radial basis function

networks for phoneme recognition, one needs to utilize a clus-

tering technique to obtain the centers of basis functions [3].

Researchers have mainly considered k-means clustering ap-

proach for this purpose [2, 3]. K-means clustering approach

applies the randomly generated points as the initial centers of

clusters and updates the position of the centers at every iter-

ation. The updating process of this approach indicates that

it only searches the narrow surrounding of the initial centers.

This may cause the algorithm to converge to suboptimal solu-

tions. In contrast, PSO is less sensitive to effect of the initial

conditions due to its population-based nature. Thus, it can

lead to near optimal solutions. Xiang et al. [4] have used

hybrid PSO and self-organizing maps (SOM) to construct a

scheme for gene clustering. In their method, the weights are

first trained using SOM and then, they are optimized by PSO.

Also, Cui et al. [5] have proposed a new method aimed to

obtain the centers of the clusters for document clustering. In

their work, each particle is a candidate for clusters centers

and PSO is used to find the optimal centers of clusters in the

search space based on the average distance of all documentsto the cluster center. Omran et al. [6] have applied PSO to

image clustering. Their proposed approach is similar to the

previous one, but the main difference is in the definition of

fitness function. They tend to cluster images such that intra-

cluster distance and quantization error is minimized while the

distance between clusters is maximized. Multiple swarms and

cooperative swarms have been also introduced to solve opti-

mization problems [7]. Multiple swarms perform better than a

single swarm in high dimensions due to exponential increase

in the volume of the search space. When the dimensionality

of the data is high and the number of clusters is large, as in

many speech applications, the ability of a single swarm is not

sufficient to search all the solution space. Instead, multiplecooperative swarms can be considered to determine clusters

centers.

In the following section, we introduce particle swarm op-

timization. Next, we provide the proposed clustering approach

based on multiple swarms. We then present experimental re-

sults and compare the proposed approach with other cluster-

ing approaches.

2. PARTICLE SWARM OPTIMIZATION

PSO is a search method imitates swarming behaviors of flocks

of birds and schools of fish, introduced by Kennedy and Eber-

hart [8]. PSO procedure employs a swarm of individuals

known as particles to solve an optimization problem. Simi-

lar to genetic algorithms (GAs), it starts from an initial pop-

ulation and explores the search space through a number of

iterations to reach near optimal solution. Each particle i is

characterized by a position-vector xi and a velocity-vector

vi. There is a best personal position for each particle, xbpi .

6061-4244-1198-X/07/$25.00 2007 IEEE SSP 2007


2/5

Furthermore, there exist a global best among the swarm, xi .

A new velocity and position of each particle for time step t+1is obtained by

vt+1

i = wvti + c1r1(x

bp,ti x

ti) + c2r2(x

,ti x

ti), (1)

xt+1

i = xti + v

t+1i , (2)

respectively, where w is a parameter to control the impact of

the previous history of velocities on the current one, and r1, r2are two random independent variables uniformly distributed

in interval [0, 1]. Also, c1 and c2 are positive constants knownas cognitive and social components, respectively. The best

personal position of a particle i is updated as

xbp,t+1i =

xt+1

i if f(xt+1i ) f(x

bp,ti ),

xbp,ti otherwise,

(3)

where f(xti) denotes the fitness value of particle i at iterationt. The global best of the swarm is also updated as

x,t+1 = arg minxt

i

{f(xt1),...,f(xtn))}. (4)

Let us assume Y = {y1, y2,...,yK} be a set of data pointsaimed to be clustered into Kseparate clusters, whereyk is the

data forming cluster k. Each particle x = (m1, ...,mK) con-tains candidates for all clusters centers, where mk shows the

center of cluster k. To model the clustering problem as an

optimization problem, it is necessary to formulate the objec-

tive function. It can be modeled using clusters compactness

and separation measures. Compactness measure is related to

within-cluster distance. Considering m1, ...,mK as the cen-

ters of clusters 1,...,K, this measure is obtained by

Fc(m1, ...,mK) =

1

K[Kk=1

(1

nk

nkj=1

dist(mk,ykj ))], (5)

where dist(.) stands for the distance between cluster centermk and data point ykj . Separation measure shows how much

the clusters are far from each other. The clusters separation

measure can be computed as

Fs(m1, ...,m

K) =1

K(K 1)[Kj=1

Kk=j+1

dist(mj,mk)]. (6)

Hence, we deal with a multi-objective function with two dif-

ferent objectives. The former objective should be minimized,whereas the latter needs to be maximized. One common method

applied in multi-objective function optimization is to trans-

form the problem into a single objective function by consid-

ering the weighted sum of the objectives. Therefore, the new

combined minimization objective function can be defined as

F1(m1, ...,m

K) = w1Fc(m1, ...,m

K) w2Fs(m1, ...,m

K),(7)

where w1 and w2 are weights of within-cluster and between-

cluster distances such that w1 + w2 = 1. One can now runPSO to obtain the solution of the clustering problem.

Single swarm-based clustering is not sensitive to the ini-

tial points and therefore it is more probable to converge to the

vicinity of optimal solution. However, its performance de-

grades when the dimensionality of feature space or the num-

ber of clusters increases. We have proposed a multiple swarms-

based clustering approach in which the search space is dis-

tributed among multiple swarm and each swarm cooperates

with other swarms to get the cluster centers.

3. MULTIPLE SWARMS-BASED CLUSTERING

APPROACH

In this section, we propose a new approach for data cluster-

ing on the basis of multiple swarms. In this approach, each

swarm corresponds to one cluster and is responsible for find-

ing its corresponding clusters center. The particles within

each swarm are candidates for the corresponding clusters cen-

ter. Also, each swarm knows the global best particle of the

other swarms.

The whole procedure to reach near optimal solution in the

proposed approach is performed through two main phases:

initialization and exploration. The schematic presentation of

multiple swarms during the two phases is given in Fig. 1.

In initialization phase, the search starts from random points.

At the beginning of this phase, there are overlaps between

swarms (Fig. 1.a), whereas at the end each swarm will deal

with only a part of the search space, called swarm region (Fig.

1.b). The swarm region k is characterized by its center and

width, which are zk and , Rk, respectively. The first parame-

ter shows the center of the swarm region and the second one

gives its corresponding width. The main goal of the initializa-

tion phase is to determine these parameters for all swarms.

In this phase, a super-swarm first searches for the cen-

ter of all swarms region using single swarm-based approach.

Thereby, the positions and velocities are iteratively updated

and the global best of the super-swarm which is indeed the

center of each swarm region, is determined. The initializa-

tion phase ends when the centers of the swarm regions do

not change over successive iterations or maximum number of

iterations achieved. At the end of this phase, the width of

each swarm region needs to be calculated. To determine the

width, we propose using eigen decomposition theorem. As-

sume kmax denotes the square root of the biggest eigen value

of data points belonging to swarm k. Then, the width of each

swarm region k is then computed by

Rk = kmax , k [1,...,K], (8)

where is a positive constant selected such that the appropri-

ate coverage of search space is obtained.

Exploration phase starts when the swarms are initialized.

In this phase, there exist no super-swarm, but rather there is

607


3/5

Swarm 1

Super swarm

Swarm 2

Swarm 3Swarm 4

Swarm 3

Super swarm

Swarm 2

Swarm 1

Swarm 4

Swarm 3

Swarm 2

Swarm 1

Swarm 4

Swarm 3

Swarm 2

Swarm 1

Swarm 4

(a) Before initialization phase (b) The end of initialization phase (c) During exploration phase (d) After exploration phase

Fig. 1. Schematic presentation of multiple swarms during initialization and exploration phases. In initialization phase, there is

a super-swarm which guides the other swarms. Situation of swarms at the beginning and end of initialization phase is given in

parts (a) and (b), respectively. In the exploration phase, parts (c) and (d), there is cooperation between swarms and each swarm

knows the global best of other swarms.

an information exchange among swarms. Each swarm coop-erates with other swarms by knowing the global best of the

other swarms.

This phase contains a number of iterations converging on

the near optimal solution. Each iteration comprises two main

steps: search and make decision. In the search step, search

within each swarm region proceeds. Each swarm tries to min-

imize the following objective function

f(xki ) = w1fc(xki ) w2fs(x

ki ), (9)

where the first term, fc(xki ), indicates the within-cluster dis-

tance and is defined by

fc(xki ) =

1

nk[

nkj=1

dist(xki ,ykj )], (10)

where, xki is the particle i of swarm k and nk denotes the

number of samples of cluster k.

The second term in equation (9), fs(xki ), stands for between-

cluster distance and is defined by

fs(xki ) =

1

K 1[Kk=1

dist(xji ,mk)]. (11)

The within-cluster distance should be minimized, whereas

the distance from other clusters should be maximized. The

within-cluster distance deals with compactness of the cluster

and between-cluster shows how far that particular cluster is

from other clusters.

After defining the objective function, the mathematical

model of the clustering -in terms of the optimization prob-

lem using multiple swarms- can be defined. In the search step

within each swarm k, particles attempt to minimize the fol-

lowing optimization problem

min f(xki ) = w1fc(xki ) w2fs(x

ki ),

s.t. : xki zk Rk,(12)

where . stands for Euclidean norm. In this equation, theconstraint forces the particles of each swarm to search within

its corresponding neighborhood.

The search using multiple swarms is performed in a serial

scheme. It begins in the first swarm region and a new can-

didate for the cluster center is obtained using equation (12).

Considering this new candidate, the next swarm searches for anew candidate for its corresponding cluster center. Similarly,

this procedure is repeated for each of the following swarms to

obtain new candidates for the center of all clusters.

When the search for all swarms is completed, it is neces-

sary to make decision on the new candidates for the cluster

centers. In other words, a final decision is made on accept-

ing or rejecting the solution proposed by multiple swarms. If

the fitness value obtained by equation (7) for new candidates

for the cluster centers is smaller than the fitness value for the

former centers the new solution is valid and accepted; other-

wise, it is rejected. The algorithm of the proposed approach

is provided in Algorithm 1.

The multiple cooperative swarms approach proposed here

distributes the search task among several swarms. Therefore,

each swarm is responsible for searching a part of the solution

space, whereas in single-swarm-based approach this task is

performed by a single swarm. For high dimensional data and

in the cases where there is a high number of clusters, a single

swarm is not capable of searching the whole solution space

effectively, because the volume of search space increases ex-

ponentially as the dimension of the problem grows.

608


4/5

Algorithm 1 Multiple swarms-based clustering

Phase 1: Complete the initialization using super swarm.

Determine swarms centers z1, z2...zK .

Determine swarms widths R1,R2...RK .

Phase 2: Exploration

Step 1: Search within each swarm

1.1. Update positions and velocities.

1.2. Calculate the fitness value of new position

of all particles using equation (9).

1.3. Choose the position minimizing optimiza-

tion problem (12) and denote it as new candi-

date for corresponding clusters center.

Step 2: Make decision

2.1. Compute fitness value of new candidates

for centers of clusters using equation (7).

2.2. If fitness value is smaller than previous

iteration, accept solution; otherwise, reject it.

2.3. If termination criterion is achieved, stop;

otherwise, go step 1.

4. SIMULATION RESULTS

We have done experiments on four phonemes /aa/, /ae/, /ay/,

and /el/ from TIMIT database [9]. We have used 12 mel-

cepstral frequency coefficients (MFCCs) and speech data has

been sampled at 16 kHz. Also, the parameters of the particle

swarm algorithms have been set to w = 1.2(decreasing grad-ually), c1 = 2, c2 = 1, n = 10, w1 = 0.85, and = 1.9.

The convergence of the multiple swarms-based cluster-

ing approach in terms of fitness value is provided in Fig. 2.

According to Fig. 2, a significant improvement is observed in

the proposed approach in terms of fitness value right after the

initialization phase, when the cooperation between swarms

begins.

In Fig. 3, the convergence of the proposed clustering ap-

proach is compared with single swarm-based clustering as

well as k-means, k-harmonic means, and fuzzy c-means clus-

tering approaches in 30 independent runs. As shown in Fig 3,

the convergence of both k-means and k-harmonic means clus-

tering approaches occurs earlier that single and multiple swarms-

based clustering approaches. However, PSO-based clustering

approaches provide better results in terms of fitness value.

We have also compared the performance of the proposed

approach with other clustering approaches in Table 1. This

0 10 20 30 40 50 60 70 80 901.5

2

2.5

3

3.5

4

Iterations

Fitness

+

end of initialization phase

Fig. 2. Convergence of the multiple swarms-based clustering

approach. The simulation is repeated for 30 runs. The average

of fitness value () and the corresponding standard deviation

() are shown in the figure.

0 20 40 60 80 100

2

2.5

3

3.5

4

Iterations

Fitness

KHarmnic means

Fuzzy cmeans

Single swarm

Multiple swarms

Kmeans

Fig. 3. Compare the convergence of the proposed approach

with other approaches

comparison is based on compactness, separation and fitness

value measures over 30 independent runs. According to Table

1, proposed approach outperforms the other approaches since

it considers multiple objectives in the model, uses multiple

cooperating swarms and distributes the search space among

multiple swarms.

We have also investigated the sensitivity of the multiple

and single swarm-based clustering approaches to the dimen-

sionality of feature space and the number of clusters. Fig. 4

presents the behavior of both single and multiple swarm-based

clustering approaches in terms of fitness value with regard to

the dimension of feature space.

As it is clear from Fig. 4, multiple swarms-based cluster-

ing approach is less sensitive to the dimensionality of feature

space in comparison with single swarm-based clustering ap-

proach. Similarly, the behavior of both approaches in terms

of fitness value with regard to the number of clusters is shown

in Fig. 5. Again, the results provided in Fig. 5 clarifies that

609


5/5

Table 1. Comparing proposed approach with others in terms

ofcompactness, separation and fitness value

Method Compactness Separation Fitness

K-means 2.41e03 22.6 5.54 0.2 2.53 0.07K-harmonic

means

2.94e03 0.1 5.640.01 2.150.01

Fuzzyc-means

3.56e030.11 0.46 0.2 2.530.07

Single

swarm

3.6e03277.7 18.170.6 1.9 0.11

Proposed 2.41e0341.3 18.1o.51 1.750.06

0 2 4 6 8 10 120.5

0

0.5

1

1.5

2

2.5

Dimensionality (d)

Fitness

(F1

)

Multiple swarms

Single swarm

Fig. 4. The behavior of multiple swarm-based and single

swarm-based clustering approaches in terms of fitness value

with regard to the dimension of feature space

1 2 3 41.5

2

2.5

3

3.5

4

4.5

Number of clusters (K)

Fitness

(F

1

)

Multiple swarms

Single swarm

Fig. 5. The behavior of multiple and single swarm-based clus-

tering approaches in terms of fitness value with regard to the

number of clusters

multiple swarms-based clustering is able to find better solu-

tions than single swarm-based clustering in terms of fitness

value as the number of clusters increases.

5. CONCLUSIONS

In this paper, we introduced a new approach for clustering

phoneme data. The idea origins from the biological behavior

of multiple swarms looking for food located in several places.

Clustering problems usually contain several objectives to be

optimized. Proposed approach is capable of considering sev-

eral objective functions simultaneously. By considering mul-

tiple swarms, the proposed approach assigns a portion of the

search space to each swarm. This strategy boosts the explo-

ration ability of the clustering approach as each swarm coop-

erating with other swarms, deals only with a portion of search

space. The proposed approach is applied on standard TIMIT

database. The results confirm that the proposed clustering ap-

proach leads to a better solution as the dimensionality of fea-

ture space or the number of clusters increases.

6. REFERENCES

[1] L. Deng and D. OShaughnessy, Speech processing: a

dynamic and optimization-oriented approach, Marcel

Dekker Inc., 2003.

[2] M. Kamel A. Ahmadi, F. Karray, Modular-based clas-

sifier for phoneme recognition, in IEEE International

Symposium on Signal Processing and Information Tech-

nology, August 2006, pp. 583588.

[3] L. Mesbahi and A. Benyetto, Continuous speech recog-

nition by adaptive temporal radial basis function, in

IEEE International Conference on Systems, Man and Cy-

bernetics, 2004, pp. 574579.

[4] X. Xiao; E.R. Dow; R. Eberhart; Z.B. Miled; R.J. Oppelt,

Gene clustering using self-organizing maps and parti-

cle swarm optimization, in IEEE Procedings of Interna-

tional Parallel Processing Symposium, 2003, p. 10pp.

[5] Xiaohui Cui; Jinzhu Gaoa and Thomas E. Potok, A

flocking based algorithm for document clustering anal-

ysis, Journal of Systems Architecture, vol. 52, no. 8-9,

pp. 505515, August-September 2006.

[6] M.G.H. Omran ; A. Salman and A.P. Engelbrecht, Dy-

namic clustering using particle swarm optimization with

application in image segmentation, Pattern Analysis and

Applications, vol. 6, pp. 332 344, 2006.

[7] F. van den Bergh and A.P. Engelbrecht, A cooperative

approach to particle swarm optimization, IEEE Transac-

tions on Evolutionary Computing, vol. 8, no. 3, pp. 225

239, June 2004.

[8] J. Kennedy and R.C. Engelbrech, Particle swarm opti-

mization, in IEEE International Conference on Neural

Networks, 1995, vol. 4, pp. 19421948.

[9] National Institute of Standards and Technology Speech

Disc 1-1.1, NTIS Order No. PB91-5050651996, Timit

acoustic-phonetic continuous speech corpus, 1990.

610

Cooperative Swarms for Clustering Phoneme Data

Documents

Transcript of Cooperative Swarms for Clustering Phoneme Data