Data Science for Network Security - LVEE• Developed by Farsight Security for transporting network...
26
Data Science for Network Security Dmitry Orekhov
Transcript of Data Science for Network Security - LVEE• Developed by Farsight Security for transporting network...
Data Sciencefor
Network Security
Dmitry Orekhov
Collecting Data
Internet
Data Flow
AnalyticsTelemetry
Traffic
Alerts,Actions
Internet
BA
C
Sensors: Vantage
Host
Services
Network
OSI
Syslogs
Specific logs
Traffic
Sensors: Domain
• NetFlow is a traffic summarization standard developed by Cisco Systems and originally used for network services billing.
• The heart of NetFlow is the concept of a flow, which is an approximation of a TCP session.
• Plenty of Open Source implementations
Transport: NetFlow
Nmsg
DNSQr
DNS Payload
Transport: Nmsg
• Developed by Farsight Security for transporting network packets, particulary -DNS
• Low-latency and compactness
• Support binary and presentation forms; protobuf for protocols
• Open impelentation (GNU)
SDN Controller
OpenFlowPacket-In
Network
Трафик
Свич SDN
Transport: OpenFlow
Use Case
DNS Tunneling
?
DNS
Question
Answer
Question
Answer
DNS Message: Big Picture
DNS
Header
Question
Answer
QName
Resource Record
Resource Record
Name
Name
Base32
Here you are: Tunnel
DNS Packet DNS Packet DNS Packet
TCP Packet
...
TCP Packet
TCP Packet
TCP Packet
TCP Packet
TCP Packet
SSH Tunnel
Discovery methods
Payload analysis
• Size of request and response• Entropy of hostnames• Statistical Analysis• Uncommon Record Types
Traffic Analysis
• Volume of DNS traffic per IP address
• Volume of DNS traffic per domain• Number of hostnames per domain• Geographic location of DNS server• Domain history• Orphan DNS requests
Effective Intrusion Detection
DNS Message
Header
Question
Answer
QName
Resource Record
Resource Record
Name
Name
... needs deep packet inspection
... and
… low-latency
DNS Message
DNS Message
DNS Message
Stream StreamStream Stream
Solution: Lambda Architecture
Online modelStreaming Processing (Spark Streaming)
Batch processing (MapReduce)
Transformation
Transformation
TransformationStream
Stream
Alert
AlgorithmsIncremental algorithmsOutlier Detection• Median Absolute Deviation, MAD• Standard Deviation from average• Standard Deviation from Moving AverageПотоковая классификация• Incremental decision tree• Hoeffding Tree (VFDT)• Half-Spaсe Trees
Offline modelStreaming Processing (Spark Streaming)
Batch Processing (MapReduce)
FilterStream
Stream
Stream
Stre
am
Alert
Offline Model - Алгоритмы
Analyze entire data set at onceHypothesis tests• Simple outlier detection far a period• Statistical criteria• Kolmogorov-Smirnov testDecision TreesAuto Regressive (AR) Moving Average (MA)
Data
Storage(Hive
or Cassandra)
ArchitectureSpark Streaming
Parsing Enrichment
Raw
Online Algorithms Alert Rules
Alerts
Kafka
Alert
TopicSpark/MapReduce
Batch
Network Appliance
Stream
Traffic
Topic
?
