A NOVEL APPROACH FOR MINING EMERGING PATTERNS IN DATA STREAMS

Hamad Alhammady

Etisalat University College - UAE hamad@euc.ac.ae

ABSTRACT

Streaming data mining is one of the most difficult tasks in Knowledge Discovery in Databases (KDD). This task is essential in many applications such as financial applications, network monitoring, marketing and others. In this model, data arrives in multiple, continuous, rapid, time-varying data streams. These characteristics make it infeasible for traditional mining techniques to deal with data streams. In this paper, we propose a new approach for mining emerging patterns [EPs] in streaming data. EPs are those itemsets whose frequencies in one class are significantly higher than their frequencies in the other classes. We experimentally prove that our new method for mining EPs has an excellent impact on the process of classifying data streams.

1. INTRODUCTION

Many recent studies show that the major challenge in streaming data is its unbounded size [1] [2] [3]. This makes it infeasible to store the entire data on disk. There are two important problems arising from this fact. Firstly, multi-pass algorithms, which need the entire data to be stored in conventional relations, cannot deal directly with data streams. Secondly, obtaining the exact answers from data streams is too expensive [4].

EPs are a new kind of patterns introduced recently [5]. They have been proved to have a great impact in many applications [6] [7] [8] [9] [10]. EPs can capture significant changes between datasets. They are defined as itemsets whose supports increase significantly from one class to another. The discriminating power of EPs can be measured by their growth rates. The growth rate of an EP is the ratio of its support in a certain class over that in another class. Usually the discriminating power of an EP is proportional to its growth rate.

For example, the Mushroom dataset, from the UCI Machine Learning Repository [11], contains a large number of EPs between the poisonous and the edible mushroom classes. Table 1 shows two examples of these EPs. These two EPs consist of 3 items. e1 is an EP from the poisonous mushroom class to the edible mushroom class. It never exists in the poisonous mushroom class,

and exists in 63.9% of the instances in the edible mushroom class; hence, its growth rate is ∞ (63.9 / 0). It has a very high predictive power to contrast edible mushrooms against poisonous mushrooms. On the other hand, e2 is an EP from the edible mushroom class to the poisonous mushroom class. It exists in 3.8% of the instances in the edible mushroom class, and in 81.4% of the instances in the poisonous mushroom class; hence, its growth rate is 21.4 (81.4 / 3.8). It has a high predictive power to contrast poisonous mushrooms against edible mushrooms.

Table1. Examples of emerging patterns.

Current approaches for mining EPs [12] [13] cannot

be used directly in data streams because they are based on multi-pass algorithms. Work in [4] introduces a new type of EPs, approximate EPs (AEPs). This type of EPs enables current mining techniques to operate on streaming data. AEPs and the AEP-tree method have shown a good accuracy in classifying streaming data. In this paper, we propose another new type of EPs, matching EPs (MEPs). These MEPs has two advantages over AEPs in terms of mining complexity and classification accuracy (details are discussed later).

2. RELATED WORK

The data stream model differs from the conventional stored relation model in a number of ways. The most significant difference is that data streams are unbounded in size. Most of the instances from a data stream have to be discarded after processing. However, a certain number of instances can be stored for future analysis. This number is proportional to the available memory. That is, the data stream model does not preclude the presence of some data stored in conventional relations [1].

The idea of storing some instances from a data stream in conventional relations is fundamental to many techniques used in the data stream model. These techniques include sliding windows, sampling, and synopsis data structures. They are the basic features of

EP Support in poisonous mushrooms

Support in edible mushrooms

Growth rate

e1 0% 63.9% ∞ e2 81.4% 3.8% 21.4

e1 = {(ODOR = none), (GILL_SIZE = broad), (RING_NUMBER = one)} e2 = {(BRUISES = no), (GILL_SPACING = close), (VEIL_COLOR = white)}

1-4244-0779-6/07/$20.00 ©2007 IEEE

any Data Stream Management System (DSMS) such as STREAM [14].

Sliding windows [1] have a noticeable power to obtain approximate answers to data stream queries. This technique involves using a sliding window of recent data from the data stream rather than operating over the entire range of data. For example, if an unlabeled instance arrives from a data stream, and it needs to be classified to one of the classes associated with this data stream, then, only a certain number of recent instances (a window) will be used to train the classifier.

Figure 1. Sliding window technique

Figure 1 sketches the idea behind this technique. The sliding window technique has the advantage of being well-defined. In addition, it is a deterministic method which avoids the problem of bad approximation caused by random sampling. Most essentially, it accentuates recent data which is considered to be the most interesting data in a large number of the real-life applications [1]. However, the sliding window technique suffers from the elimination of some important information contained in old (discarded) data. That is, sliding windows do not represent the whole range of knowledge contained in the data, but rather a portion (proportional to the size of the sliding window) of that knowledge. This problem may affect the quality of approximation.

Sampling [15] is another technique for approximation in the data stream model. In this case, the streaming data is randomly sampled to a certain number of instances. This number is proportional to the available memory. In contrast with the sliding window technique, sampling has the advantage of representing the whole range of old data. The representation level is proportional to the sampling rate. On the other hand, sampling may suffer from problems caused by noisy instances being selected during the random sampling process.

Synopsis data structures [1] aim at summarizing the most important characteristics of the whole range of data. These important characteristics play a key role in classifying future unlabeled instances. Synopsis data structures, like the sampling technique, have the advantage of representing the whole range of old data. Moreover, these structures avoid the problem caused by noisy instances. The reason is that they store the important characteristics rather than the data itself. EPs can be thought of as synopsis data structures because they represent the discriminating characteristics of the data they are related to.

Approximate emerging patterns (AEPs) adopt approximation to mine EPs from data streams [4]. Mining AEPs is based on mining EPs from blocks of streaming data and merging the resulting EP sets to get a

fixed number of AEPs. These special EPs are described as approximate because they are not mined from the complete range of data. The AEP tree is a new type of decision trees to classify streaming data. This tree uses AEPs rather than data instances to make decisions on the classes of unlabelled data.

3. EMERGING PATTERNS AND CLASSIFICATION

Let obj = {a1, a2, a3, ... an} be a data object following the schema {A1, A2, A3, ... An}. A1, A2, A3.... An are called attributes, and a1, a2, a3, ... an are values related to these attributes. We call each pair (attribute, value) an item.

Let I denote the set of all items in an encoding dataset D. Itemsets are subsets of I. We say an instance Y contains an itemset X, if X ⊆ Y.

Definition 1. Given a dataset D, and an itemset X, the support of X in D, sD(X), is defined as

||)(

)(D

XcountXs D

D = (1)

where countD(X) is the number of instances in D containing X.

Definition 2. Given two different classes of datasets D1 and D2. Let si(X) denote the support of the itemset X in the dataset Di. The growth rate of an itemset X from D1 to D2, )(

21Xgr DD → , is defined as

≠=∞==

=→

otherwise ,)()(

0)( and 0)( if ,0)( and 0)( if ,0

)(

1

2

21

21

21

XsXs

XsXsXsXs

Xgr DD

(2)

Definition 3. Given a growth rate threshold ρ >1, an

itemset X is said to be a ρ -emerging pattern ( ρ -EP or

simply EP) from D1 to D2 if ρ≥→ )(21

Xgr DD .

Let C = {c1, … ck} is a set of class labels. A training dataset is a set of data objects such that, for each object obj, there exists a class label cobj ∈ C associated with it. A classifier is a function from attributes {A1, A2, A3, ... An} to class labels {c1, … ck}, that assigns class labels to unseen examples.

4. MINING MATCHING EMERGING PATTERNS

We adopt the streaming data model presented in [4]. This model is shown in figure 2. Assume that the data stream consists of two classes; C1 and C2. Data is received in blocks of size N, where N is decided according to the memory available in the system. Bt,j is a block of instances related to class j (C1 or C2) at time t.

After receiving and processing a number of data blocks, we need to gain information to classify the future unlabeled instances in the data streams. This information

Sliding Window

Past Data (Discarded) Recent Data

Future Data

can be expressed as EPs. However, mining EPs from a dataset requires the availability of all instances in this dataset. This is infeasible in data streams as data is arriving continuously.

Figure 2. Data stream model

Our method is based on mining EPs from selected blocks of data and matching these EPs with the future data. As data is streaming in blocks of size N, blocks of all classes related to period t (Bt,1 and Bt,2) are stored, processed and then discarded. EPs for both classes are mined from data blocks Bt,1 and Bt,2 before discarding them. These EPs are EPt,1 (for C1 ) and EPt,2 (for C2 ). These EPs are moved to new sets of EPs called matching EPs, MEPs. That is, MEP1 represents the MEPs of class C1 and MEP2 represents the MEPs of class C2 .

In the following stage, new data blocks arrive, Bt+1,1 and Bt+1,2. EPs are not mined from these new blocks. Instead, current MEPs are matched with the new data instances to check if they are still EPs for the new data. If MEPs retain their EP characteristics (existing in one class more than the other) they remain in the set of MEPs. Otherwise, they are eliminated from the set. For example, EPs in MEP1 are matched with the new data blocks. If they exist in Bt+1,1 more frequently than in Bt+1,2 then they are still valid EPs and remain in MEP1 otherwise they are eliminated. The same is applied to MEP2.

The above process continues for the future blocks of data until the number of EPs in MEP1 (or MEP2) is reduced to a predefined number, α , because of the elimination process. In this case, EPs are mined from the new blocks of data and best EPs are chosen to refill MEP1 (or MEP2) set. For example, suppose that the number of EPs in MEP1 has been reduced to α at time t+8, then, EPs are mined from block Bt+9,1 to create its set of EPs, EPt+9,1. The strongest1 EPs in EPt+9,1 are used to refill MEP1 again. Algorithm 1 explains the idea of mining MEPs.

We call the emerging patterns resulted from the above algorithm matching emerging patterns (MEPs). These EPs are mined from selected blocks of data rather than the complete range of data. In spite of that, our approach guarantees that these MEPs are inherited from all the old discarded data. That is, for each class, we have to store its limited set of MEPs rather than its growing number of data instances.

MEPs are motivated by the following points:

1 The strength of an EP is proportional to its support and growth rate.

1. If an old EP does not exist any more in the future blocks of data it is eliminated.

2. If an EP exists in the future data blocks, it remains in the MEP set.

Algorithm 1. Mining MEPs from streaming data

MEP1= Φ , MEP2= Φ , t = 0 As data is streaming Do t = t + 1 If EPs in MEP1 < α EPt,1 = mined EPs from Bt,1 MEP1 = MEP1 U strongest EPs in EPt,1 Else For each EP e in MEP1 Match e with Bt,1 and Bt,2 If e is no more an EP Remove e from MEP1 End if End for End if If EPs in MEP2 < α EPt,2 = mined EPs from Bt,2 MEP2 = MEP2 U strongest EPs in EPt,2 Else For each EP e in MEP2 Match e with Bt,2 and Bt,1 If e is no more an EP Remove e from MEP2 End if End for End if End do

The above two points support the importance of the

recent data which is the main advantage of the sliding window technique. Moreover, they prove that the MEPs are related to all the previous data which is the advantage of the sampling technique. Furthermore, MEPs overcome the problem of mining EPs from all blocks of streaming data in the AEP method by applying a selective approach to choose certain blocks of data. This ensures that the mining process is conducted when necessary only. Our approach ensures that at each period of time we have limited sets of MEPs that best represents all the discarded data. These sets can be used at any time to classify unlabeled data instances using the AEP-tree proposed in [4].

5. EXPERIMENTAL EVALUATION

In this section, we apply five techniques to the data streaming model described in section 4. These techniques are AEP-tree using MEPs, AEP-tree using AEPs, random sampling, sliding window, and a traditional classifier (C4.5 decision tree). Beside applying it alone, the C4.5 decision tree will be the base classifier for the random sampling, and sliding window techniques.

The testing method is 10-fold-cross validation. This method is adapted to agree with the data stream model adopted in our experiments. The data is divided into ten

B1,1 B1,2

B2,1 B2,2 B3,1 B3,2

B4,1 B4,2

N

Data Stream

Class 1 Class 2

folds. For each round of the 10-fold-cross validation, one fold is used for testing and the other nine folds are used for training. The training folds act as the blocks of data explained in the data stream model.

Table 2 shows the performance of the previous techniques on 10 real-life data from the UCI repository [11]. The last row in the table indicates the average accuracy of each technique. The results show that our proposed method, AEP-tree using MEPs, has the highest accuracy average. It outperforms the sliding window and sampling techniques on all datasets. It also outperforms the AEP-tree using AEPs on 8 datasets. This indicates that our proposed method for mining MEPs is capable of gaining accurate knowledge from streaming data.

Table 2. Experimental results

Dataset C4.5* Sliding window sampling

AEP tree

(AEPs)

AEP tree (MEPs)

Breast 94.6 74.1 70.4 83.4 81.7 Cleve 73.8 59.4 55.8 65.1 69.6 Diabetes 73.4 59.8 55.2 62.9 67.3 Heart 80.6 61.7 59.4 71.5 75.1 labor 76.9 52.3 50.7 63.3 69.9 CC 85.3 70.2 67.6 77.6 75.1 Hayes-roth 70.2 60.3 55.2 66.5 68.1 Hepatitis 81.8 70.4 67.3 75.1 78.1 Horse 85.2 72.6 70.8 81.4 82.2 Segment 93.5 81.9 81.4 87.8 88.8 Average 81.53 66.27 63.38 73.46 75.59

* C4.5 with complete knowledge

6. CONCLUSIONS

In this paper, we study the mining of emerging patterns in data streams by introducing a special type of emerging patterns, matching emerging pattern (MEPs). This type of EPs can be easily mined from data streams by applying a selective approach to conduct the mining process.Our experiments prove that MEPs is capable of gaining important information from streaming data. This information increases the accuracy of classification.Our research opens a wide avenue for the applications of emerging patterns (EPs). EPs can now be used in different data stream applications. Our future work will focus on designing new techniques for mining EPs from data streams. These techniques might be useful for mining other types of patterns which are currently infeasible in the data stream model.

REFERENCES

[1] B. Babcock, S. Babu, M. Datar, R. Motwani, and J. Widom. Models and Issues in Data Stream Systems. In Proceedings of the 21st ACM Symposium on Principles of Database Systems (PODS’02), Madison, Wisconsin, USA. [2] G. Dong, J. Han, L.V.S. Lakshmanan, J. Pei, H. Wang and P.S. Yu. Online Mining of Changes from Data Streams: Research Problems and Preliminary Results. In Proceedings of the 2003 ACM SIGMOD Workshop on Management and Processing of Data Streams, San Diego, CA, USA.

[3] M. Garofalakis, J. Gehrke, and R. Rastogi. Querying and Mining Data Streams: You Only Get One Look. In Proceedings of the 28th International Conference on Very Large Data Bases (VLDB’02), Hong Kong, China. [4] Alhammady, H., & Ramamohanarao, K. (2005). Mining emerging patterns and classification in data streams. In Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence (WI), Compiegne, France, pp. 272-275. [5] G. Dong, and J. Li. Efficient Mining of Emerging Patterns: Discovering Trends and Differences. In Proceedings of the 1999 International Conference on Knowledge Discovery and Data Mining (KDD'99), San Diego, CA, USA. [6] H. Alhammady, and K. Ramamohanarao. The Application of Emerging Patterns for Improving the Quality of Rare-class Classification. In Proceedings of the 2004 Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD'04), Sydney, Australia. [7] H. Alhammady, and K. Ramamohanarao. Using Emerging Patterns and Decision Trees in Rare-class Classification. In Proceedings of the 2004 IEEE International Conference on Data Mining (ICDM'04), Brighton, UK. [8] H. Alhammady, and K. Ramamohanarao. Expanding the Training Data Space Using Emerging Patterns and Genetic Methods. In Proceeding of the 2005 SIAM International Conference on Data Mining (SDM’05), New Port Beach, CA, USA. [9] H. Fan, and K. Ramamohanarao. A Bayesian Approach to Use Emerging Patterns for Classification. In Proceedings of the 14th Australasian Database Conference (ADC’03), Adelaide, Australia. [10] Guozhu D., Xiuzhen Z., Limsoon W., and Jinyan L.. CAEP: Classification by Aggregating Emerging Patterns. In Proceedings of the 2nd International Conference on Discovery Science (DS'99), Tokyo, Japan. [11] C. Blake, E. Keogh, and C. J. Merz. UCI repository of machine learning databases. Department of Information and Computer Science, University of California at Irvine, CA, 1999. [12] H. Fan, and K. Ramamohanarao. An Efficient Single-Scan Algorithm For Mining Essential Jumping Emerging Patterns for Classification. In Proceedings of the 2002 Pacific-Asia Conference on Knowledge Discovery and Data Mining, Taipei, Taiwan. [13] H. Fan, and K. Ramamohanarao. Efficiently Mining Interesting Emerging Patterns. In Proceedings of the 4th International Conference on Web-Age Information Management (WAIM’03), Chengdu, China. [14] Stanford Stream Data Management (STREAM) Project. http://www-db.stanford.edu/stream [15] B. Babcock, M. Datar, and R. Motwani. Sampling From a Moving Window Over Streaming Data. In Proceedings of the 2002 Annual ACM-SIAM Symposium On Discrete Algorithms, San Francisco, CA, USA.