System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A...
Transcript of System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A...
![Page 1: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/1.jpg)
![Page 2: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/2.jpg)
2
![Page 3: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/3.jpg)
3
System Event LogExist practically on
every computer system!
Automatic Analysis?
![Page 4: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/4.jpg)
4
SystemEventLog
Started service A on port 80Executor updated: app-1 is now LOADING……
![Page 5: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/5.jpg)
5
SystemEventLog
Structured Data
Log key
printf(“Started service %s on port %d”, x, y);
LOG PARSING
Started service A on port 80Executor updated: app-1 is now LOADING……
Started service * on port *Executor updated: * is now LOADING……
![Page 6: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/6.jpg)
6
SystemEventLog
Structured Data Anomaly Detection
LOG ANALYSIS
LOG PARSING
Started service A on port 80Executor updated: app-1 is now LOADING……
Started service * on port *Executor updated: * is now LOADING……
Log key
printf(“Started service %s on port %d”, x, y);
![Page 7: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/7.jpg)
7
![Page 8: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/8.jpg)
8
SPELLA streaming log
parser published in ICDM’16
Deletion of file1 complete. Deletion of file1 complete.
log keylog message
Deletion of file2 complete. Deletion of * complete.
parameters
[ ]
[file2]
![Page 9: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/9.jpg)
9
DeepLog
Anomaly Detection Diagnosis
![Page 10: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/10.jpg)
10
TrainingStage
DetectionStage
MODELS
![Page 11: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/11.jpg)
11
DetectionStage
MODELS
![Page 12: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/12.jpg)
12
DetectionStage
MODELS
![Page 13: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/13.jpg)
13
DetectionStage
MODELS
![Page 14: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/14.jpg)
14
DetectionStage
MODELS
![Page 15: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/15.jpg)
15
TrainingStage
MODELS
![Page 16: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/16.jpg)
16
TrainingStage
MODELS
![Page 17: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/17.jpg)
17
TrainingStage
MODELS
![Page 18: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/18.jpg)
18
TrainingStage
MODELS
![Page 19: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/19.jpg)
19
TrainingStage
MODELS
![Page 20: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/20.jpg)
20
TrainingStage
MODELS
![Page 21: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/21.jpg)
21
Use long short-term memory (LSTM) architecture
In detection stage, DeepLog checks if the actual next log key is among its top g probable predictions.
![Page 22: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/22.jpg)
22
Method 1: Using LSTM prediction probabilities
Method 2: Using co-occurrence matrix
![Page 23: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/23.jpg)
23
![Page 24: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/24.jpg)
24
Multi-variate time series data anomaly detection problem!--- Leverage LSTM to check reconstruction error.
![Page 25: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/25.jpg)
25
Evaluation results on HDFS log data. (over a million log entries with labeled anomalies)
PCA (SOSP’09), IM (UsenixATC’10), N-gram (baseline language model)
![Page 26: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/26.jpg)
26
Evaluation results on Blue Gene/L log, with and without online model update.
![Page 27: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/27.jpg)
27
Evaluation results on OpenStack cloud log with different confidence intervals (CIs)
![Page 28: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/28.jpg)
28
Diagnosis using constructed workflow.
Injected anomaly: during VM creation, network speed from controller to compute node is throttled.
![Page 29: System Event Log - School of Computingmind/papers/deepLog_short.pdf · 2017-12-04 · 29 DeepLog A realtime system log anomaly detection framework. LSTM is used to model system execution](https://reader031.fdocuments.in/reader031/viewer/2022041912/5e681ecb40b70344f144a19d/html5/thumbnails/29.jpg)
29
DeepLog
➢ A realtime system log anomaly detection framework.
➢ LSTM is used to model system execution paths and log parameter values.
➢ Workflow models are built to help anomaly diagnosis.
➢ It supports online model update.