Mining Dynamics of Data Streams in Multidimensional Space ...
Mining High-Speed Data Streams
description
Transcript of Mining High-Speed Data Streams
MINING HIGH-SPEED DATA STREAMS
Presented by:
Yumou Wang
Dongyun Zhang
Hao Zhou
INTRODUCTION The world’s information is doubling
every two years. From 2006 to 2011, the amount of
information grew by a factor of 9 in just five years.
INTRODUCTION By 2020 the world will generate 50
times the amount of information and 75 times the number of "information containers"
However, IT staff to manage it will grow less than 1.5 times.
Current algorithms can only deal with small amount of data less than a day’s data of many applications.
For example, banks, telecommunication companies.
INTRODUCTION Problems : When new examples arrive at a
higher rate than they can be mined, the amount of unused data grows without bounds as time progresses.
Today, to deal with these huge amount of data in a responsible way is very important.
Mining these continuous data streams brings unique opportunities, but also new challenges.
BACKGROUNDDesign Criteria for mining High
Speed Data Streams It must be able to build a model using at
most one scan of the data. It must use only a fixed amount of main
memory. It must require small constant time per
record.
BACKGROUND Usually, use KDD system to operate
this examples when they arrive.Shortcomings: learning model
learned are highly sensitive to example ordering compare to the batch model.
Others can produce the same model as batch version but very slower.
CLASSIFICATION METHOD Input: Examples of the form (x,y), y is the class
label, x is the vector of attributes. Output: A model y=f(x), predict the classes y of
future examples x with high accuracy.
DECISION TREE One of the most effective
and widely-used classification methods.
A decision tree is a decision support tool that uses a tree-like graph or model.
Decision trees are commonly used in machine learning.
BUILDING A DECISION TREE 1. Starting at the root. 2. Testing all the attributes and choose
the best one according to some heuristic measure.
3. Split one node into branches and leaves.
4. Recursively replacing leaves by test nodes.
EXAMPLE OF DECISION TREE
EXAMPLE OF DECISION TREE
PROBLEMS There are some problems existed in
traditional decision tree. Some of them assume that all training data
examples can be stored simultaneously in main memory.
Disadvantages: Limited the number of examples can be learned from.
Disk-based decision tree learners: examples in disk, repeatedly reading them.
Disadvantages: expensive when learning complex trees.
HOEFFDING TREES Designed for extremely large datasets Main idea: To find the best attribute at
a given node by considering only a small subset of the training examples that pass through the node.
Using how many examples is sufficient
HOEFFDING BOUND
n
In
2
)1(R 2
Definition: The statistical result that can decide how many examples “n” using by each node is called Hoeffding bound.
Assume: R—the range of variable r n independent observations mean: r’
With probability 1-δ, the true mean of r is at least r’-є
HOEFFDING BOUND
n
In
2
)1(R 2
This function is a decreasing function n is bigger, the є is smaller It is the difference between true value and
mean value of r.
HOEFFDING TREE ALGORITHM
HOEFFDING TREE ALGORITHM Inputs:
S -> is a sequence of examples,X -> is a set of discrete attributes,G(.) -> is a split evaluation
function, δ -> is one minus the desired
probability of choosing the correct attribute at any given node.
Outputs: HT -> is a decision tree.
HOEFFDING TREE ALGORITHMGoal: Ensure that, with a high probability, the attribute chosen using n examples, is the same as that would be chosen using infinite examples.
Let Xa be the attribute with the highest observed G’ and Xb be with second highest attribute.After seeing n examples.
Let ΔG’ = G’(Xa) – G’(Xb)ΔG’ > ϵ
Thus a node needs to accumulate examples from the stream until ϵ becomes smaller than ΔG.
HOEFFDING TREE ALGORITHM The algorithm constructs the tree using
the same procedure as ID3. It calculates the information gain for the attributes and determines the best attributes.
At each node it checks for condition ΔG > ϵ. If the condition is satisfied, then it creates child nodes based on the test at the node.
If not it streams in more training examples and carries out the calculations till it satisfies the condition.
HOEFFDING TREE ALGORITHMMemory cost d—number of attributes c—number of classes v—number of values per attribute l—number of leaves in the tree The memory cost for each leaf is
O(dvc) The memory cost for whole tree is
O(ldvc)
ADVANTAGES OF HOEFFDING TREE
1. Can deal with extremely large datasets.
2. Each example to be read at most once in a small constant time. Makes it possible to mine online data sources.
3. Build very complex trees with acceptable computational cost.
VFDT—VERY FAST DECISION TREE
Breaking ties Reduce waste Useful under condition where
Use of Split may not change with a single example Significantly reduce the time of re-computation
Memory cleanup Measurement of Clearance of least promising leaves Option of enabling reactivation
VFDT—VERY FAST DECISION TREE
Filtering out poor attributes Dropping early Reduces memory consumption
Initialization Can be initialized with other existing tree Set a head start
Rescans
TESTS—CONFIGURATION
14 Concepts Generated by random decision trees using Number of leaves: 2.2k to 61k Noise level: 0 to 30%
50k examples for testing Available memory: 40MB Legacy processors
TESTS—SYNTHETIC DATA
, ,
TESTS—SYNTHETIC DATA
TESTS—SYNTHETIC DATA
TESTS—SYNTHETIC DATA
TESTS—SYNTHETIC DATA
TESTS—SYNTHETIC DATA
Time consumption20m examples
VFDT takes 5752s to read, 625s to process
100k examplesC4.5 takes 36sVFDT takes 47s
TESTS—PARAMETERS
W/ & w/o over-pruning
TESTS—PARAMETERS
W/ ties vs. w/o ties65 nodes vs. 8k nodes for VFDT805 nodes vs. 8k nodes for VFDT-boot72.9% vs. 86.9% for VFDT83.3% vs. 88.5% for VFDT-boot
vs. VFDT: +1.1% accuracy, +3.8x timeVFDT-boot: -0.9% accuracy, +3.7x time5% more nodes
TESTS—PARAMETERS
40MB vs. 80MB memory7.8k more nodesVFDT: +3.0% accuracyVFDT-boot: +3.2% accuracy
vs. 30% less nodesVFDT: +2.3% accuracyVFDT-boot: +1.0% accuracy
TESTS—WEB DATA
For predicting accesses
1.89m examples
61.1% with most common class
276230 examples for testing
TESTS—WEB DATA
Decision dump 64.2% accuracy 1277s to learn
C4.5 with 40MB memory 74.5k examples 2975s to learn 73.3% accuracy
VFDT-bootstrapped with C4.5 1.61m examples 1450s to learn after initialization(983s to read)
TESTS—WEB DATA
MINING TIME-CHANGING DATA STREAMS
WHY IS VFDT NOT ENOUGH?
VFDT, assume training data is a sample drawn from stationary distribution.
•Most large databases or data streams violate this assumption –Concept Drift: data is generated by a time-
changing concept function, e.g. •Seasonal effects •Economic cycles
•Goal: –Mining continuously changing data streams –Scale well
WHY IS VFDT NOT ENOUGH?
Common Approach: when a new example arrives, reapply a traditional learner to a sliding window of w most recent examples
–Sensitive to window size •If w is small relative to the concept shift rate,
assure the availability of a model reflecting the current concept
•Too small w may lead to insufficient examples to learn the concept
–If examples arrive at a rapid rate or the concept changes quickly, the computational cost of reapplying a learner may be prohibitively high.
CVFDT
CVFDT (Concept-adapting Very Fast Decision Tree learner) –Extend VFDT –Maintain VFDT’s speed and accuracy –Detect and respond to changes in the example-
generating process
CVFDT (CONTD.) With a time-changing concept, the current
splitting attribute of some nodes may not be the best anymore.
An out dated subtree may still be better than the best single leaf, particularly if it is near the root. – Grow an alternative subtree with the new best
attribute at its root, when the old attribute seems out-of-date.
Periodically use a bunch of samples to evaluate qualities of trees. – Replace the old subtree when the alternate one
becomes more accurate.
HOW CVFDT WORKS
EXAMPLE
SAMPLE EXPERIMENT RESULT
CONCLUSION AND FUTURE WORK
CVFDT is able to maintain a decision-tree up-to—date with a window of examples by using a small constant amount of time for each new examples that arrives.
Empirical studies show that CVFDT is effectively able to keep its model up-to-date with a massive data stream even in the face of large and frequent concept shifts.
Future Work: Currently CVFDT discards subtrees that are out-of-date, but some concepts change periodically and these subtrees may become useful again – identifying these situations and taking advantage of them is another area for further study.
THANK YOU