Post on 16-Apr-2017
1
SAMOA: A Platform for Mining Big Data Streams
Nicolas KourtellisAssociate Researcher
Telefonica I+D, Barcelona
SAMOA: Scalable Advanced Machine Online Analysis
2
What is Big Data?Search queriesFacebook postsEmailsTweetsPhoto sharesClicks on ads…
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
3
How BIG is your data?Volume (+ Variety)
Too large for RAM of single commodity serverVelocity
Too fast for CPU of single commodity server
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
4
What is the Streaming Paradigm?High amount of data, high speed of arrivalUpdated models at “real” timePotentially infinite sequence of dataChange over time (concept drift)
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
5
Mining Big Data StreamsApproximation algorithms:
Single pass, one data item at a timeSub-linear space and time per data itemSmall error with high probability
A platform solution:Support different algorithms & processing enginesDistributedScalable
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
6
What is SAMOA?Scalable Advanced Massive Online AnalysisA platform for mining big data streams
Framework for developing new distributed stream mining algorithms
Framework for deploying algorithms on new distributed stream processing engines
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
7
Taxonomy
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
8
SAMOA ArchitectureMachine LearningAlgorithms
Distributed StreamProcessing Engines Flink
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
9
Why is SAMOA important?Program once, run everywhere
Reuse existing infrastructureAvoid deploy cycles
No system downtimeNo complex backup/update processNo need to select update frequency
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
10
ML Developer API
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
11
ML Developer API
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
12
Deployment
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
13
Easy to get!
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
14
Easy to get!
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
15
Easy to get!
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
16
Easy to test!bin/samoa storm target/SAMOA-Storm-0.3.0-SNAPSHOT.jar"PrequentialEvaluation-d /tmp/dump.csv-i 1000000 -f 100000-l (classifiers.trees.VerticalHoeffdingTree -p 4 -k)-s (generators.RandomTreeGenerator –r 1 -c 2 -o 10 -u 10)"
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
17
Case study: Decision TreesVHT: Vertical Hoeffding Tree*
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
Task parallelism
*VHT: Vertical Hoeffding Tree. N. Kourtellis, G. De Francisci Morales, A. Bifet, A. Mordupo. IEEE BigData 2016.
18
Case study: VHT
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
Horizontal Parallelism
19
Case study: VHT
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
Vertical Parallelism
20
Benefits of Vertical ParallelismHigh number of attributes:
high level parallelism (e.g., documents)vs. task parallelism:
obvious parallelism observedvs. horizontal parallelism:
reduced memory usage (no model replication)parallelized split computation
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
21
Vertical Hoeffding Tree
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
22
Preliminary results: TweetsZipf skew: 1.5Bag of words: 100, 1000, 10000 (attributes)Size of tweet: ~15 wordsInstances: 1,000,000Class: positive or negative (Gaussian random variable)
10 runsLocal vs. Storm virtual cluster
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
23
Results: Accuracy
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
4 8 16 local0
20
40
60
80
100
Classification Accuracy vs.Parallelism Level vs.Number of Attributes
100 words1000 words10000 words
Parallelism Level
Cor
rect
Cla
ssifi
catio
n %
24
Results: Speedup
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
4 8 16 local0
1
2
3
4
5
Speedup vs.Parallelism Level vs.Number of Attributes
100 words1000 words10000 words
Parallelism Level
Spee
dup
25
Is SAMOA for you?Are you dealing with:
Big fast data?Possibly endless streams of data?Evolving data?
Do you need updated models at real time?Do you want to test an algorithm on different DSPEs?
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
26
SAMOA Team
Albert Bifet
GianmarcoDe Francisci Morales
Nicolas Kourtellis
Matthieu Morel
Arinto Murdopo
Olivier Van Laere
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
27
StatusApache Incubator
Released version 0.3.0 in JulyExecution Engines
Input: Local FS HDFS Kafka [pending]
Heron?
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
28
Algorithms in SAMOAExisting:
Vertical Hoeffding Tree (classification) CluStream (clustering) Adaptive Model Rules (regression)
Pending: Distributed Naïve Bayes Stochastic Gradient Descent Adaptive + Boosting VHT Parallelized Gradient Boosted Decision Tree PARMA (frequent pattern mining) …
Check Samoa Roadmap for more
Looking for contributors!
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15
29
SAMOA: A Platform for Mining Big Data Streams
@ApacheSAMOAhttp://samoa.incubator.apache.org/
https://github.com/apache/incubator-samoa
Nicolas Kourtellis@kourtellis
nicolas.kourtellis@telefonica.com
SAMOA: Scalable Advanced Machine Online Analysis 30/09/15