SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity...
Transcript of SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity...
![Page 1: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/1.jpg)
SESSION ID:
#RSAC
Ravi Honnavalli
FIM AND SYSTEM CALL AUDITING AT SCALE IN A LARGE CONTAINER DEPLOYMENT
CSV-R14
Staff EngineerWalmartTwitter handle: @ravi_honnavalli
![Page 2: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/2.jpg)
#RSAC
Disclaimer
2
NOTE: All content discussed here are out of self learning and not related to the work I do at Walmart.
![Page 3: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/3.jpg)
#RSAC
Ever increasing amount of logs
3
![Page 4: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/4.jpg)
#RSAC
Overwhelming amount of choices
4
Too many options!!
Static rules
?
ML?
Event source
?
Agent vs Agentles
s
SIEM tools
![Page 5: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/5.jpg)
#RSAC
Flood of OSS tools
5
Elasticsearch
osquery
journald ElastAlert
TensorFlowUnstructured datastores
fluentd
![Page 6: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/6.jpg)
#RSAC
GOAL: Demystifying the choices we have
6
•Classifying event sources
•Understand event source type
•Understand how they work in containers
Understanding types of event sources
•Understand the insights we are looking for
•Build a stack based on the event classification
•If needed customize existing open source tools
•Build adaptors / tools that join the whole chain
Build our own stack based on insight needed
• The stacks discussed in this presentation are by no means the only stack availableMake an informed
decision
![Page 7: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/7.jpg)
#RSAC
• kauditd• Inotify
Kernel
Possibilities of tools evaluated
7
• auditd• go-auditd• go-audit-container• osquery
Data shippers• Logstash• Filebeat• Fluentd
Sink• File• Syslog
Deployment tools• Chef• Puppet• Ansible
Fleet manager• Zentral• Kolide• Doorman• Hand crafted tool
Unstructured data stores• Elasticsearch• MongoDB
Graphing and reporting• Kibana• Grafana• ElastAlert• Custom tools based on
query DSL/Lucene
Data Preprocessing• Custom tool to build a
training set
ML• TensorFlow• Layer depth• Optimization
algorithm• Learning rate• Gradient descent
mechanisms
![Page 8: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/8.jpg)
#RSAC
Classification of event sources
8
Event sources
Event based
syscall inotify
Scheduled query
agent
![Page 9: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/9.jpg)
#RSAC
Security insight based on event source type
9
sysc
all Looking for specific
outliers among mostly normal
dynamic events.
• Like identifying outliers
• Monitoring constantly for a specific malicious system call along with other criteria (uid, etc)
ino
tify Safe-guarding specific
sensitive files / area in the file system
• Watch for CREATE/ACCESS/MODIFY/DELETE events on specific files
agen
t Scheduled activities for static information
• OS patch level queries, vulnerable kernel modules, mis-configuration
![Page 10: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/10.jpg)
#RSAC
SYSTEM CALLS
![Page 11: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/11.jpg)
#RSAC
Why syscall?
11
Fundamental transit points between user land
and kernel
Every process makes system calls disclosing information of
its activity
Several user space tools that send audit information
(auditd, go-audit, go-audit-container)
Can provide deep insight when aggregated and drilled
down
Ideal candidate to build a machine learning training set as the volume of data is huge
![Page 12: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/12.jpg)
#RSAC
Audit component
12
User land
Kernel land
Kauditd
Syscall interfaceNetlinksocket
Reporting daemon
User space application
Namespacingconcern!
![Page 13: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/13.jpg)
#RSAC
Container anatomy
13
PID FS net
UTC UID RPC
Container 1
PID FS net
UTC UID RPC
Container 2
PID FS net
UTC UID RPC
Container 3
Audit, syslog, kernel key ring
Kernel
![Page 14: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/14.jpg)
#RSAC
go-audit-container
14
![Page 15: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/15.jpg)
#RSAC
Audit log to gain insights at scale
User land
Kauditd
Syscall interfaceNetlinksocket
go-audit-container
User space application
Elasticsearch
Grafana
Kibana
Pre-Processsink
Pagerduty
Slack
TensorFlow
Kernel
![Page 16: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/16.jpg)
#RSAC
Demo: Privilege escalation
16
![Page 17: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/17.jpg)
#RSAC
INOTIFY
![Page 18: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/18.jpg)
#RSAC
Inotify component
18
User land
Kernel landInotify component
User space applicationInotify_add_watch
Watch list
Inotify_event { }
Inotify_event { }
Inotify_event { }
Inotify_event { }
Event queue
inotify sysctlsare not
namespaced!
![Page 19: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/19.jpg)
#RSAC
Why inotify?
19
Lesser CPU consumption on
an average
Missing details in the reports
![Page 20: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/20.jpg)
#RSAC
inotify based stack for FIM
User land
Kernel land
Elasticsearch
Grafana
Kibana
Pre-Processsink
Pagerduty
Slack
TensorFlow
osquery
Inotifycomponent
Register watch
Notify event
![Page 21: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/21.jpg)
#RSAC
AGENTS
![Page 22: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/22.jpg)
#RSAC
osquery
22
osquery osqueryosqueryosquery
Fleet Manager
OS OSOS OS
Distributed Query
![Page 23: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/23.jpg)
#RSAC
Osquery stack to get insights at scale
Elasticsearch
Grafana
Kibana
Pre-Processsink
Pagerduty
Slack
TensorFlow
osquery
OS
Extension plugin
Fleet manager
![Page 24: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/24.jpg)
#RSAC
Compliance query
24
![Page 25: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/25.jpg)
#RSAC
East West Threat
25
![Page 26: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/26.jpg)
#RSAC
osquery packs
26
![Page 27: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/27.jpg)
#RSAC
Learning from using fixed queries in Kibana, Grafana and custom tools
27
Robust rules need a lot of
queries
Fixed queries need to be constantly updated for new
patterns
Any small variation of the rules is a false negative
Machine learning helps improve our ability to detect
anomalies and broadly classify security posture
![Page 28: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/28.jpg)
#RSAC
MACHINE LEARNING
![Page 29: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/29.jpg)
#RSAC
High level differences
29
Unsupervised
Elasticsearch ML
Anomaly detection
Time series data
Supervised
Pre-processink-osquery
Explicit labelling and pre-processing
Explicit data classification on disparate info
![Page 30: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/30.jpg)
#RSAC
Picture credit: https://unsplash.com/@ripato30
Elasticsearch ML: Detecting outliers
![Page 31: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/31.jpg)
#RSAC
Demo of ElasticSearch ML
31
![Page 32: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/32.jpg)
#RSAC
Use case: Supervised Learning
32
Classifying data from different event sources
Broad classification into RED/YELLOW/GREEN
Classifying to have a big picture of the security posture of the organization
![Page 33: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/33.jpg)
#RSAC
Supervised learning: Building a training set is key
33
ElasticsearchPreprocessink-
osqueryLabelling TensorFlow
![Page 34: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/34.jpg)
#RSAC
Pre-processink-osquery stages
Stage 0• Query one probe at a time from ES
• Label into RED/YELLOW/GREEN
• Write to stage_0.csv
Stage 1 • Merge into existing stage_1.csv
Manual Labeling
• At this stage human administrator can manually label events that were not labeled or which were incorrectly labeled by the automated rules
Stage 2• Transform into numeric
values, normalizing and mean centering
![Page 35: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/35.jpg)
#RSAC
ML choices if you are building your own solution
35
Activation function
Learning rate
Depth of the network
Batch size vs iterations
Optimzers
![Page 36: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/36.jpg)
#RSAC
Results of our experiment
36
ReLU activation
Batch Size
Adam optimizer
Lower the learning rate the better
![Page 37: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/37.jpg)
#RSAC
Lessons learnt
37
Identify event sources keeping in mind nature of events needed, frequency, memory and CPU footprint of the agents on the application nodes, etc.
Be aware of how and what is namespaced in containers when configuring event sources in containers.
High level classification of ML
Machine learning helps in getting intelligent insights, but needs training and fine tuning.
![Page 38: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/38.jpg)
#RSAC
Apply
38
Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure
Grow the solution to more detailed monitoring. Keep in mind load on the infrastructure, when thousands of nodes start sending events.
Try applying ML based on unsupervised learning and then get your hands dirty with training
Think of possibilities outside of what is discussed here today
![Page 39: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/39.jpg)
#RSAC
AuditNG suite
39
https://github.com/auditNG/preprocessink-osquery
https://github.com/auditNG/go-audit-container
![Page 40: SESSION ID: CSV-R14 FIM AND SYSTEM CALL AUDITING AT …...Start with a simple File Integrity monitoring implementation using inotify log. Observe load of FIM events on the infrastructure](https://reader033.fdocuments.in/reader033/viewer/2022041504/5e23b6fb87f71367a069cc37/html5/thumbnails/40.jpg)
#RSAC
Questions?
40
You can also reach out later:Twitter handle: @ravi_HonnavalliLinkedIn: https://www.linkedin.com/in/ravi-honnavalli-0535163/