Post on 28-Dec-2015
Best Known Methods in Security Events Correlation
Mohammed Fadzil Haron GSEC GCIA
April 12, 2005
2 IT@Intel
Agenda
Correlation overviewKnowledge requirementsMethodologyData representationReaction
3 IT@Intel
Correlation defined
A relation existing between phenomena or things or between mathematical or statistical variables which tend to vary, be associated, or occur together in a way not expected on the basis of chance alone…[1]
[1] http://www.webster.com
4 IT@Intel
Overview Correlation is the next security big thing in
importance An important tool in the security analyst’s toolbox
for monitoring security events To be most effective, most – if not all – events
should be examined Defense in depth means more data from different
technologies, vendors, and products Huge amount of data to analyze; terabytes in size
and growing Reduce false-positive and false-negative findings
compared to use of a single product/technology Expensive manned 24x7 monitoring capabilities
5 IT@Intel
Ultimate goal
Et = Dt + Rt Exposure time (Et): The time the resource,
information, or organization is susceptible to attack or compromise.
Detection time (Dt): The time it takes for the vulnerability or the threat to be detected.
Reaction time (Rt): The time it takes for the individual, group, or organization to respond and eliminate or mediate the vulnerability or risk.
“Time Based Security” by Winn Schwartau
6 IT@Intel
Security events flow
7 IT@Intel
Axiom on correlation
1. You only see the tip of the iceberg
2. Know the environment and perimeter of defense well
3. Don’t trust the tool; trust your judgment4. “Automate whenever possible” [1]
5. Use the simplest data representation possible
6. Balance between over-correlated and under-correlated
7. Get the big picture8. “The truth is in the packet” [1]
[1] Toby Kohlenberg, Intel Corp.
8 IT@Intel
Knowledge requirements
Know your environmentKnow your perimeter of defenseAutomate tasksSimplify data representation
9 IT@Intel
Know your environment
Knowing the ins and outs of your network is a necessity
– External network, DMZ and internal network architecture– Other networks, such as VPN and dial-up– Logistical and geographical locations of servers and users– Different operation systems, applications and functionality
of servers and client machines– Network switches and routers in use– Logistical and geographical locations of critical servers
(DNS, WINS, DHCP) as well as high-valued servers (web servers, servers containing intellectual properties)
– You cannot know everything yourself, so know the individual experts on each piece of the network puzzle
10 IT@Intel
Example of environment knowledge usage Can isolate IP addresses of Internet, DMZ and
internal network for different categorization– Potential detection of external attack versus inside job
VPN and dial-up services introduce other threats and need to be given separate consideration
Allows assignment of customized severity levels for different services, such as DNS and servers housing intellectual property, for upgraded security needs
11 IT@Intel
Source of events
Host level – Syslog, HIDS/HIPS, eventlog, log files, apps logs, anti-virus signature level
Network level – NIDS/NIPS, NBAD, firewall, network routers and switch logs, active directory logs, VPN logs, third-party authentication logs
Audit – Vulnerability scanning, OS and patch level
Knowledgebase – Software vulnerabilities and exploits
12 IT@Intel
Know your perimeter of defense
FirewallIDSIPSAudit capabilitiesHost level defensesPENSVulnerability scanning dataAnd so on.
13 IT@Intel
Know your firewalls
Location – Outer-facing, inner-facing, DMZ, internal, internal isolated network
Type – Packet filter, stateful, application firewall/proxy
What’s allowed versus deniedCapabilities versus shortcomings
14 IT@Intel
Know your IDS/IPS
Which product deployed? NIDS, HIDS/HIPS, NIPS
Where were they deployed? What kind of traffic is being monitored?
What product/vendor deployed?Capabilities versus shortcomings
15 IT@Intel
Know your audit capabilities
Where are logs being kept? Syslog server or logs on host?
How long have logs being kept? Rotated? Know your syslog servers
16 IT@Intel
Host level defenses
Anti-virus logsMinimum security specification compliance
enforcement software logsOS, service packs, patches-level
information
17 IT@Intel
Automate tasks as much as possible
Daunting tasks to detect intrusion due to:– Amount of data involved reaching terabyte range
– Complexity of network environment architecture with Internet presence, DMZ, WAN, MAN, PAN, LAN, VOIP, VPN, Dial-up
– Complexity of perimeter of defense
– Large IP address ranges used internally, that is, using Class A 10.x.x.x
– Multiple internally isolated networks with different type of policies, and access controls
18 IT@Intel
What and where to automate
Data aggregation – at data source and event manager
Manual, repetitive tasks – at event manager and reaction
Data correlation – event managerSimplify data representation – event
manager consoleIncident notification – event manager
19 IT@Intel
Group your assets
Break down IP addresses into groups, such as internal, DMZ and others for Internet
Determine and group all critical servers, such as DNS, WINS, and DHCP
Determine and group all high valued servers, such as file shares, web servers, and FTP servers, and encrypted content servers for intellectual properties
20 IT@Intel
Types of correlation
Sets– String a group of events together to generate a
trigger
Sequences– String a group of events together in sequence or
particular order to generate a trigger
Statistical– Deviation of normal behavior, such as mean or
normal curve
21 IT@Intel
Methods of correlation Rule
– Manually constructed, easy to create/update. Usually explicit in nature and can be applied to set, sequence and threshold types. Contains three elements: condition, time interval, and response.
Heuristic– Similar to anti-virus signature. One signature can detect multiple
variations. More implicit than explicit in nature, thus potential for higher false positives/negatives.
Fuzzy Logic / Artificial Intelligence– Model approach to correlation that can dynamically adapt to
changing environment. Difficult to produce and still immature; very cutting-edge.
Hybrid– No one doing them all yet. Commonly used are heuristic and
rule.
22 IT@Intel
Correlation constraint
Time– Time should be considered when creating time
box correlation
– Correct time is critical in correlation
– Time synchronization is crucial
Context– Order of events sequence is important
– Context can be necessary in correlation rules
23 IT@Intel
Sample of correlation flow
INTERNETExternal
attacker’s IP address
Outer IDS detection
NO YES
Outer FirewallOuter Firewall Accept/Deny
Deny Accept AcceptDeny
DMZ IDS detection
DMZ IDS detection
NO YES
Inner Firewall
Inner Firewall
Inner Firewall
Inner FirewallDeny
Deny Deny DenyAccept
AcceptAccept
Accept
NO YES
Inner IDS detection
Inner IDS detection
Inner IDS detection
Inner IDS detection
NO NONOYES YES YES YESNO
24 IT@Intel
Graphical representation
Seeing is believingPros
– Can represent huge data in simple and easy to understand graphs
Cons– Not many tools (commercial/open source) with
this capability
– If exist, limited capabilities
25 IT@Intel
Effective graphics should…
Show the dataAvoid distorting dataPresent a large volume of data in small
spaceMake large data sets coherentShow several levels of detailProvide clear purpose of data presentationRepresent the data and not the underlying
technology, methodology, and design
26 IT@Intel
Forms of data representation
GraphsLink graphChartsData mapsTime seriesNarrative graphics (space and time)AnimationVisualizationVirtual reality
27 IT@Intel
Scanning graph(One source to many target relationship)
Mar 14 08:33:20 66.34.244.12:2827 -> xxx.yyy.1.1:18905 SYN ******S*
Mar 14 08:33:20 66.34.244.12:2830 -> xxx.yyy.1.2:18905 SYN ******S*
Mar 14 08:33:20 66.34.244.12:2833 -> xxx.yyy.1.3:18905 SYN ******S*
Mar 14 08:33:22 66.34.244.12:2836 -> xxx.yyy.1.4:18905 SYN ******S*
Mar 14 08:33:22 66.34.244.12:2839 -> xxx.yyy.1.5:18905 SYN ******S*
Mar 14 08:33:22 66.34.244.12:2842 -> xxx.yyy.1.6:18905 SYN ******S*
Mar 14 08:33:22 66.34.244.12:2845 -> xxx.yyy.1.7:18905 SYN ******S*
Mar 14 08:33:20 66.34.244.12:2848 -> xxx.yyy.1.8:18905 SYN ******S*
Harder to internalize
Scan activity easily recognized
28 IT@Intel
Link graph
Stage 1 of worm
propagation
29 IT@Intel
Link graph
Stage 2 of worm
propagation
30 IT@Intel
Link graph
Stage 3 of worm
propagation
31 IT@Intel
Moving average(Simple network anomaly detection)
0
20
40
60
80
100
120
140
160
180
1 2 3 4 5 6 7 8 9
Monitored Events
Moving Average
Increase in moving average, showing an increase in activities
Example: Monitoring port 445
32 IT@Intel
Animation movie
Inbound connection attempts to San Diego State University (SDSU) from external source (unauthorized)
Representing 332 GB of raw data, 3.4 billion raw syslog records, and 1 million events
Period of 1996-2002 (6 years) Available at
http://security.sdsc.edu/probes-animations/index.shtml
33 IT@Intel
Animation movie
34 IT@Intel
Reaction to correlated data
Enforcement for malware cleaningBlocking to minimize malware propagation
and attackInvestigation for malicious non-worm
activitiesLearning mode for improving data (reducing
false-positives and false-negatives)
35 IT@Intel
Conclusion
Correlation is a must tool for information security professionals
Time saved in detection will allow faster response time
Faster response time will minimize damages to your assets
36 IT@Intel
Questions?
37 IT@Intel
References
Event correlation; http://www.computerworld.com/networkingtopics/networking/management/story/0,10801,83396,00.html
“Protecting the Enterprise with Scalable Security Event Management, Part II - Intelligent Event Correlation”; Michael Mychalczuk; https://www.sans.org/webcasts/show.php?webcastid=90468
“Thinking about Security Monitoring and Event Correlation“; http://www.securityfocus.com/infocus/1231