Big data expo - machine learning in the elastic stack
-
Upload
bigdataexpo -
Category
Data & Analytics
-
view
401 -
download
0
Transcript of Big data expo - machine learning in the elastic stack
![Page 1: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/1.jpg)
Dr Stephen Dodson, Machine Learning, Elastic
Machine Learning in the Elastic Stack
![Page 2: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/2.jpg)
2
Elastic Stack
Beats
Logstash
Kibana
Elasticsearch
X-PackX-Pack X-Pack
Security
Alerting
Monitoring
Reporting
Graph
Machine Learning
![Page 3: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/3.jpg)
3
Extracting useful, valuable information is hard
![Page 4: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/4.jpg)
4
Search
Aggregations
Visualization
Machine Learning
Extracting useful, valuable information is hard
![Page 5: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/5.jpg)
Machine Learning1Algorithms and methods for data driven prediction, decision making, and modelling
Supervised Learning
Prediction based on examples of correct behavior
1Machine Learning Overview, Tommi Jaakkola, MIT
Unsupervised Learning
No explicit target, only data, goal to model/discover
Semi-supervised Learning
Supplement limited annotations with unsupervised learning
Active Learning
Learn to query the examples actually needed for learning
Transfer Learning
How to apply what you have learned from A to B
Reinforcement Learning
Learning to act, not just predict; goal to optimize the consequences of
actions
Other! …
![Page 6: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/6.jpg)
Machine Learning1Algorithms and methods for data driven prediction, decision making, and modelling
Supervised Learning
Prediction based on examples of correct behavior
1Machine Learning Overview, Tommi Jaakkola, MIT
Unsupervised Learning
No explicit target, only data, goal to model/discover
Semi-supervised Learning
Supplement limited annotations with unsupervised learning
Active Learning
Learn to query the examples actually needed for learning
Transfer Learning
How to apply what you have learned from A to B
Reinforcement Learning
Learning to act, not just predict; goal to optimize the consequences of
actions
Other! …
Time Series Anomaly Detection
![Page 7: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/7.jpg)
Machine Learning - Technical Debt
7
• Machine Learning Systems generally involved a large amount of plumbing and complexity that incur large ongoing maintenance costs
From Google Paper: Sculley, D., et al. "Machine learning - The high-interest credit card of technical debt." (2014).
![Page 8: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/8.jpg)
© Elasticsearch BV
Has my order rate dropped significantly?
![Page 9: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/9.jpg)
Has my order rate dropped significantly?
• Learn models from past behaviour (training, modelling)
• Use models to predict future behaviour (prediction)
• Use predictions to make decisions
Expected value @ 15:05 = 1859
Actual value @ 15:05 = 280
Probability = 0.0000174025
![Page 10: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/10.jpg)
Demo: Simple Time Series
![Page 11: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/11.jpg)
Has my system changed behaviour?
i-5cfd3dcb
...
…
i-f1e94994
i-ece626ff i-ebc323df
![Page 12: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/12.jpg)
Has my system changed behaviour?
i-5cfd3dcb i-f1e94994
i-ece626ff i-ebc323df
...
…
![Page 13: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/13.jpg)
Demo: Multiple Time Series
![Page 14: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/14.jpg)
Do my application logs contain unusual messages?
![Page 15: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/15.jpg)
Do my application logs contain unusual messages?Classify unstructured log messages by clustering similar messages
Nor
mal
Log
Mes
sage
sU
nusu
al lo
g M
essa
ges
![Page 16: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/16.jpg)
Demo: Log Messages
![Page 17: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/17.jpg)
17
Ingest, Enrich, Visualize, Analyse, Alert
Elasticsearch
X-pack
Master Nodes (3)
Ingest Nodes (X)
Data Nodes - Hot (X)
Data Nodes - Warm (X)
Beats
Log Files Metrics
Wire Data your(beat)
Filebeat Module
NGINX
Kibana
X-pack
Instances (X)
![Page 18: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/18.jpg)
18
Coming soon - forecasting…and more…
![Page 19: Big data expo - machine learning in the elastic stack](https://reader031.fdocuments.in/reader031/viewer/2022022415/5a6493387f8b9a57568b4c49/html5/thumbnails/19.jpg)
Exceptwhereotherwisenoted,thisworkislicensedunderhttp://creativecommons.org/licenses/by-nd/4.0/
CreativeCommonsandthedoubleCinacircleareregisteredtrademarksofCreativeCommonsintheUnitedStatesandothercountries.
Thirdpartymarksandbrandsarethepropertyoftheirrespectiveholders.
Please attribute Elastic with a link to elastic.co