Breach Detection Systems Buyers Guide

ANALYST BRIEF

Breach Detection Systems Buyer’s Guide

Authors – John W. Pirc, Francisco Artes

Overview Enterprises recognize the need for the deployment of technologies that will provide protection through the detection and subsequent identification of breaches. While intrusion detection systems (IDS) provide alerts only for attempts against the corporate network by known attacks (and a security team must still verify whether the attack was successful), breach detection systems (BDS) provide alerts for network compromise incidents from both known and unknown attacks. Whether through the use of host-‐agents or through near-‐line and in-‐line network monitors, the different BDS products that are available share a common goal of breach detection and reporting. This guide reviews the BDS features that are important to the enterprise, as identified by the research of NSS Labs.

NSS defines BDS as systems that are implemented to identify and report actual breaches as well as attempted breaches. Many BDS products on the market can identify an attempted breach by the download, or “drop,” of a file that the BDS product knows to be malware. An actual breach occurs when the downloaded file is executed and the workstation performs the activity intended by the malware. This activity may include the workstation sending communications to a command & control (C&C) server, or even polling the Domain Name System (DNS) in an attempt to contact the C&C. An attempted breach occurs when the “drop” contains a payload that is not compatible with the workstation.

NSS Labs Analyst Brief – Breach Detection Systems Buyer’s Guide

2

Current BDS Vendors NSS worked closely with participating BDS vendors during testing. The vendors listed in Figure 1 have recently announced product offerings in the BDS market. It is important to note that not all of these vendors have participated in the testing. Future iterations of the “Breach Detection Systems Buyer’s Guide” will provide an update on these vendors.

Figure 1 – BDS Vendors

NSS Labs Findings • A BDS is able to detect threats by using a network appliance or a stand-‐alone endpoint client, or by using a

combination of both of these solutions. • A BDS can identify pre-‐existing breaches as well as malware that is introduced to the system through side

channels. • A BDS that is unable to identify side-‐channel attacks or command and control traffic from infected machines in

an accurate and timely manner should be considered as little more then a network AV device. • While high-‐end network core switches and dedicated spanning devices will be capable of supporting most

BDS/IDS requirements, NSS engineers have noted significant port spanning issues with many workgroup switches.

• The BDS is the last line of defense against breaches that go undetected by current security technologies, or that are unknown by these technologies.

• Enterprises are adopting BDS in order to address the increase of targeted persistent attacks (TPAs).

Vendors

AhnLab

Check Point

Cyphort

Damballa

Fidelis

FireEye

Fortinet

Lastline

McAfee

Palo Alto Networks

Sourcefire

Trend Micro

Triumfant


3

NSS Labs Recommendations • Enterprises looking to deploy a BDS should understand that there are differences in the maturity, technology,

and scalability of the solutions that are offered by different vendors. The following points should be considered when looking to evaluate a BDS solution since they can impact deployment, scalability, and privacy issues for an enterprise:

o Does the solution require an end-‐point BDS, a network BDS, or a combination of both? o Does the solution require that data is sent to the cloud for sandboxing? o Does the solution provide the ability to turn off cloud sandboxing? o Does the solution provide the ability to customize a sandbox to an enterprise gold image?

• Customers utilizing a technology that provides sandboxes should take the time to develop and deploy virtual images of corporate desktops if supported by the BDS selected for deployment.

• Verify whether a vendor is able to detect pre-‐existing breaches as well as malware that is introduced to the system through side channels.

• Use caution when blocking either a C&C or a malicious source of the malware drop. Choosing to block an entire domain may have a negative impact on legitimate traffic; it is important to be precise.

• Verify that the network switches to which the BDS devices are connected are capable of accurate port spanning at line rates.


4

Table of Contents

Overview ................................................................................................................................ 1 Current BDS Vendors .................................................................................................................................... 2

NSS Labs Findings .................................................................................................................... 2

NSS Labs Recommendations ................................................................................................... 3

Analysis .................................................................................................................................. 6 Centralized Management ............................................................................................................................. 7 Policy Definition ........................................................................................................................................ 7 Appliance Management ........................................................................................................................... 7 Mitigation Response ................................................................................................................................. 7 Reporting .................................................................................................................................................. 7

Application Awareness ................................................................................................................................. 8 Multiple Protocols And Applications ......................................................................................................... 8 Protocol Tunneling On Standard Ports ...................................................................................................... 8

Machine (Workstation/Server) Identification .............................................................................................. 8 User/Group Identification ............................................................................................................................ 8 Policy Definition ........................................................................................................................................ 8 Attacks On Or By Specific Users Or Groups ............................................................................................... 9 Integration with Common Directories ...................................................................................................... 9

Malware Identification ................................................................................................................................. 9 IPv6 Support .............................................................................................................................................. 9 SSL/TLS Encryption .................................................................................................................................... 9 Heuristics .................................................................................................................................................. 9 DNS Heuristics ........................................................................................................................................... 9 Pattern Matching .................................................................................................................................... 10 IP Reputation .......................................................................................................................................... 10 File-‐Based (Store-‐And-‐Forward) Inspection Or Reassembly-‐Free Inspection (Network Streaming) ........ 10 Identify Zero Day/Unknown Malware .................................................................................................... 10 Detect Botnet Activity ............................................................................................................................. 10 Detection of Malware Dropper/Downloader .......................................................................................... 11 Ability to Provide Indication of Payload Execution ................................................................................. 11 Mobile Malware Detection ..................................................................................................................... 11 Domain Reputation To Identify Malicious Domains ............................................................................... 11 Ability to Identify Malware Executable Files ........................................................................................... 11 Flow Monitoring ..................................................................................................................................... 11 Content Analysis ..................................................................................................................................... 11 Sandboxing ............................................................................................................................................. 12

Performance Rating .................................................................................................................................... 12


5

Reading List .......................................................................................................................... 13

Contact Information .............................................................................................................. 14

Table of Figures Figure 1 – BDS Vendors ................................................................................................................................................. 2

Figure 2 – BDS Taxonomy ............................................................................................................................................. 6

Figure 3 – Protocol Support .......................................................................................................................................... 8


6

Analysis The highly complex BDS utilize several techniques, including local sandboxes, emulation, the capturing of network data, hosted cloud-‐based sandboxing, and definition and heuristic matching. Some BDS vendors are also utilizing endpoint agents that communicate directly with cloud-‐based services. Regardless of a vendor’s architectural approach, the main objective of a BDS is to alert the operator to a breach.

Breach detection has the ability to identify malware (or the results of a malware infection) on a corporate network. This is done in two ways.

• The monitoring of external ports and connections such as web, email, and file transfers, in order to inspect executable files that are being downloaded by workstations.

• The monitoring of outbound traffic and the identification of C&C connections and other behaviors that are indications of an infected system.

Breach detection deployments vary by vendor, but there are a few common deployment scenarios.

• Out-‐of-‐band network deployments (similar to that of an IDS) that use third-‐party taps or port spanning on a switch.

• In-‐line network deployments similar to that of an intrusion prevention system (IPS) • Endpoint deployments using client-‐based agents that are installed on each workstation and are centrally

controlled and updated via a centrally managed, most often cloud-‐based service.

While most BDS vendors claim similar features, the information contained within this analyst brief should be used for the purposes of initial product selection based on infrastructure requirements. Readers are encouraged to read the vendor data sheets and to verify them against upcoming results from the BDS testing, which will be documented in the NSS “Product Analysis Reports” (PARs) and “Comparative Analysis Reports” (CARs), and which will be available to NSS subscribers.

This guide identifies the categories that are considered an important component of every BDS solution.

Figure 2 – BDS Taxonomy


7

Centralized Management While most BDS solutions are designed as a network perimeter monitor, some of the solutions offered allow for additional monitoring for breaches at the network core and at other choke points within the network. Some vendors also provide an endpoint client to every workstation on the corporate network. For a truly functional solution and as an example of a network-‐based BDS, a customer will need to deploy sensors at each Internet ingress/egress point, and also potentially at virtual private network (VPN) links, neutral or “demilitarized zones” (DMZs), and the network core and its major switching points. These deployments demand sophisticated central management of the devices, similar to the management systems used by distributed security tools, such as IPS and firewalls.

Policy Definition

Policy definition will differ across vendor BDS platforms. The key requirement of a BDS is the detection of malware, or the effects of malware on a corporate network (subsequent cross-‐infection of other machines on the network, exfiltration of data, command and control communication, and so on). This detection can be performed in multiple stages, including but not limited to: detection of known malware website URLs; initial malware drops; privilege escalation detection through sandboxing or emulation; and C&C callbacks. BDS products are able to provide retroactive notification about malware, and some can disable malware while it is in transit. However, while such features are helpful, the primary function of a BDS product is the ability to set policy for breach detection notification, which will allow for remediation of the malware.

Appliance Management

Potentially the most important part of centralized management within BDS is application management, which includes patching, general configuration, and management of the sandbox images (where available).

Mitigation Response

A vendor’s ability to respond to malware that has been identified positively as malicious; it can be accomplished via these methods:

• TCP reset to terminate suspicious sessions. • Quarantine machines to isolated VLANS for remediation. • API integration with in-‐line security appliances such as firewalls, IPS, and so on.

Reporting

Good reporting is almost as important as the detection of malware. Since BDS generally do not actively block malware or C&C communications, it is essential that a BDS solution provides robust reporting information and that it is highly flexible regarding the times, conditions, and variables for any given report. Reports must contain information on the suspected malware, such as its payload and how it may spread, and reports must identify clearly the machine(s) that have been infected. At a minimum, the reporting capability should contain the following:

• Detailed reports that allow administrators to identify and remediate infected systems • Ability to create custom reports • Ability to integrate with SIEM platform • Ability to auto-‐generate helpdesk tickets


8

Application Awareness It is critical that BDS recognize application data and send notification when valid data is found to be using an invalid transmission protocol (for example, if the transmission protocol is not RFC compliant or when non-‐DNS traffic is being transmitted over port 53).

Multiple Protocols And Applications

Malware utilizes multiple vectors for communication and infection, often changing such channels (protocols) between malware variants. A viable BDS must be able to monitor application traffic on all common application protocol pathways. Figure 3 provides examples of common application protocol pathways:

Figure 3 – Protocol Support

Protocol Tunneling On Standard Ports

As with multiple protocols and application support, a BDS must be able to identify the misuse of an application protocol through its placement on a non-‐standard port. Examples would include tunneling a binary stream of data on TCP Port 80, which is used for HTTP. Malware will often use common outbound ports since most organizations leave these ports open to allow for effective use of the Internet by employees.

Machine (Workstation/Server) Identification

As with the identification and logging of authenticated end users, each BDS vendor’s device should be capable of performing identification, logging, and reporting based on the filters that have been set for a machine’s identification, for example, as an IP address or as a fully qualified domain name.

User/Group Identification

This feature provides contextual insight beyond an IP address by providing a username and group (for example, finance or marketing) with the ability to connect with a directory system such as lightweight directory access protocol (LDAP) or Active Directory.

Policy Definition

This is the ability to set policy based on end users or on groups contained within commonly-‐used directories.

ProtocolDNS Domain)Name)SystemFTP File)Transfer)ProtocolHTTP Hypertext)Transfer)ProtocolIMAP Internet)Message)Access)ProtocolIRC Internet)Relay)ChatNFS Network)File)SystemPOP3 Post)Office)ProtocolSMB Server)Message)BlockSMTP Simple)Mail)Transfer)Protocol


9

Attacks On Or By Specific Users Or Groups

Depending on the business use of BDS, the ability to assign users to groups and to set risk and alert thresholds based on either an individual’s or a group’s affiliation may be critical. Within a financial institution, it may be important to set a higher level of alert and priority on specific users (for example, when accounts that are used by tellers or employees with access to the financial infrastructure become infected with man-‐in-‐the-‐browser malware, such as Zeus).

Integration with Common Directories

For LDAP and Active Directory services to operate efficiently and without the need to develop a redundant directory or install workstation clients, it is important that the BDS integrates with common directory systems such as LDAP, Microsoft’s Active Directory, and terminal access controller access control system (TACACS+). The “Management Comparative Analysis Report” (CAR) provides granular information on the directories that are supported by each of the tested vendors.

Malware Identification

The following sections discuss the methods that are used by BDS for the detection of malware. Note that the claims made by vendors on their methods of identifying malware are broad and sweeping. While each product may technically offer support for these capabilities, overall efficacy has yet to be proven by independent testing. The BDS group test will test these capabilities, and the results will be documented in the “BDS Product Analysis Reports” (PARs) and the “BDS CARs,” which are available to NSS subscribers.

IPv6 Support

Currently, Internet protocol version 6 (IPv6) is not yet a common requirement worldwide, and the need for migration to it varies depending upon geographic location. Buyers should determine whether a vendor has support for IPv6 traffic detection and also whether the vendor can manage devices with an IPv6 address.

SSL/TLS Encryption

While these features were not specifically tested as part of the “Breach Detection Systems: Test Methodology 1.5,” some vendors offer the functionality to intercept, decrypt, and inspect traffic within some encrypted channels, such as SSL/TLS.

Heuristics

Heuristics are used to identify traits of malware that might otherwise go undetected. Instead of searching for defined signatures, heuristics identifies potential malware based on an understanding of malware behavior. In some cases, the delta between “normal” and “abnormal” data may be used to identify a breach (for example, it can be used to identify sudden differences in the patterns of network traffic).

DNS Heuristics

Within a BDS, domain name system (DNS) heuristics watch for the unconventional use of DNS traffic from hosts, since this may indicate a C&C search/connection. Some BDS systems also perform logic steps that look for matching between forward and reverse lookup, such as whether the domain name referenced is a misspelling of a popular site, or whether it does not use a common combination of dictionary words, or whether the registrar for


10

the domain name has been used in other malware campaigns. Answering these questions allows a BDS to identify and trigger alerts on possible phishing and malware drop sites.

Pattern Matching

Pattern matching is similar to traditional anti-‐virus detection, where files are compared to a database of known malware. Used alone, this methodology is often inadequate for the detection of variants of known malware, and it cannot detect zero day/unknown malware. In order to be effective, pattern matching requires frequent updates of the database (also known as a dictionary).

IP Reputation

Reputation identification is designed to allow vendors to whitelist or blacklist websites, or to assign them a ranked score. If a workstation accesses a website with a bad reputation, i.e., a website that is known to host malware, the system will monitor the connection and will mark any downloaded payload as having the potential to be malware.

File-‐Based (Store-‐And-‐Forward) Inspection Or Reassembly-‐Free Inspection (Network Streaming)

Store-‐and-‐forward refers to the characteristic whereby files are fully reconstructed by the inspection engine prior to being examined, after which they can be forwarded to their original destination. The result is a high degree of accuracy since the entire file is available be inspected, but the accuracy is achieved at the expense of speed.

Reassembly-‐free inspection allows files to be inspected directly from the stream of network traffic without having to fully reconstruct them. This has the advantage of speed and low memory utilization, since the entire file no longer needs to be reconstructed in memory prior to inspection. However, there is also an increased risk of false positive or false negative events, given that only portions of a file are available at any given point in the inspection process.

BDS currently tends to be a store-‐and-‐forward technology since it is typically deployed out-‐of-‐band in a passive detection mode, and thus does not need to process files in real time.

Identify Zero Day/Unknown Malware

The ability to identify the following types of malware:

• Variants of known malware that have been modified so they do not match current signatures in anti-‐virus and other anti-‐malware signatures. Many commercial malware tools are able to generate thousands of variants, which are tested against common anti-‐virus tools; those that go undetected are used in malware campaigns.

• New malware may exist in the wild for an extended period of time before it is officially detected or categorized by current signature databases.

BDS identifies this malware through technologies such as heuristics and sandboxing, and through cloud services, where thousands of BDS feeds provide the data needed to classify and flag new malware. NSS researchers tested the capabilities of the BDS under test by exposing them to new malware variants and zero day threats.

Detect Botnet Activity

In some cases the malware infection is benign: although the code is present on the machine, it requires input from the C&C server in order for it to operate within the botnet. The C&C then deploys the botnet against new targets with the aim of propagating infected hosts for the malware campaign.


11

Detection of Malware Dropper/Downloader

This is essential for each vendor since the malware dropper/downloader is important in the identification of malware on the host (for example, workstation, server, mobile device).

Ability to Provide Indication of Payload Execution

The vendor’s ability to detect that a malicious payload has been executed on the host. In some cases, this can identify “patient zero” and also provide the ability to remediate the infected host.

Mobile Malware Detection

The vendors’ ability to detect malware on a mobile device, or malware in transit that is targeting a mobile device.

Domain Reputation To Identify Malicious Domains

The vendor’s ability to identify certain domains that are associated with malware.

Ability to Identify Malware Executable Files

The first opportunity for a BDS to identify a malware infection is during the “drop,” which is the act of downloading malware to a workstation. NSS research shows that the time between when malware downloads and when it becomes active can vary between tests. The next opportunity for a BDS to identify a malware infection occurs after a “drop” has deployed its payload and the malware begins contacting Internet-‐based resources, such as websites and C&C servers. Most detection of malware by BDS occurs because the drop’s payload is a known file, or in more advanced methods, because a suspect payload is deployed to a sandbox and tested. The common protocols used to infect workstations by malware campaigns are HTTP and email. These two protocols should be mandatory; email (generally SMTP) is currently an advanced feature for BDS, but with more malware campaigns using phishing, this is a strong consideration for many enterprises.

Flow Monitoring

Flow monitoring, or the ability to recompile and analyze application data on the network, is the principal technology used in BDS. The “Breach Detection Systems Test Methodology 1.5” tests a vendor’s ability to perform this function in near-‐line mode (for example, through the use of Ethernet taps or spanned ports on switches). Some systems utilize in-‐line monitoring similar to that of an IPS; such systems are moving toward the ability to prevent malware from being dropped or from establishing a C&C.

Note that for accurate identification of malicious traffic by passive detection devices such as IDS and BDS it is essential that the network switches to which the detection devices are connected are capable of accurate port spanning at line rates. While high-‐end network core switches and dedicated spanning devices will be capable of supporting most BDS/IDS requirements, NSS engineers have noted significant port spanning issues with many workgroup switches.

Content Analysis

BDS devices should have the ability to capture suspect executable files and place them into sandboxes where their behavior may be observed. While most systems support sandbox capabilities on the local network, some use sandboxes that are hosted in the cloud. Here, it is possible to check the MD5 hash of the suspect file against the cloud’s database to determine if the file has already been cataloged or if another BDS has already submitted the file for review.


12

Sandboxing

Sandboxing is the ability to model behaviors with operating systems and web browsers in order to detect privileged escalation of a process or malicious behaviors of recently downloaded content. It is important to note that there are various methods of sandboxing. For example, sandboxing can be performed on the local BDS appliance, or on end-‐point software, or it can be cloud based (ability to send the information to the cloud for analysis).

Performance Rating Performance will vary between vendors. NSS recommends a review of the forthcoming “BDS Performance CAR” to ensure that a selected vendor is advertising accurate performance numbers within its datasheet.


13

Reading List The Targeted Persistent Attack (TPA) The Misunderstood Security Threat Every Enterprise Faces. NSS Labs https://www.nsslabs.com/reports/targeted-‐persistent-‐attack-‐tpa-‐misunderstood-‐security-‐threat-‐every-‐enterprise-‐faces

Breach Detection: Don’t Fall Prey to Targeted Attacks. NSS Labs https://www.nsslabs.com/reports/breach-‐detection-‐dont-‐fall-‐prey-‐targeted-‐attacks

Breach Detection Systems: Test Methodology 1.5. NSS Labs https://www.nsslabs.com/reports/breach-‐detection-‐systems-‐test-‐methodology-‐15

https://www.nsslabs.com/reports/targeted-persistent-attack-tpa-misunderstood-security-threat-every-enterprise-faces

https://www.nsslabs.com/reports/breach-detection-dont-fall-prey-targeted-attacks

https://www.nsslabs.com/reports/categories/methodologies


14

© 2013 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the authors.

Please note that access to or use of this report is conditioned on the following:

1. The information in this report is subject to change by NSS Labs without notice.

2. The information in this report is believed by NSS Labs to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this report are at the reader’s sole risk. NSS Labs is not liable or responsible for any damages, losses, or expenses arising from any error or omission in this report.

3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY NSS LABS. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-‐INFRINGEMENT ARE DISCLAIMED AND EXCLUDED BY NSS LABS. IN NO EVENT SHALL NSS LABS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.

4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet the reader’s expectations, requirements, needs, or specifications, or that they will operate without interruption.

5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report.

6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective owners.

Contact Information NSS Labs, Inc. 206 Wild Basin Rd Building A, Suite 200 Austin, TX 78746 USA +1 (512) 961-‐5300 [email protected] www.nsslabs.com

This analyst brief was produced as part of NSS Labs’ independent testing information services. Leading products were tested at no cost to the vendor, and NSS Labs received no vendor funding to produce this analyst brief.

Breach Detection Systems Buyers Guide

Documents

Transcript of Breach Detection Systems Buyers Guide