Research Paper CAMNEP: An intrusion detection system for ...

Special issue: The future of software engineering for security and privacy

Progress in Informatics, No. 5, pp.65–74, (2008) 65

Research Paper

CAMNEP: An intrusion detection system for high-speed networks

Martin REHAK1, Michal PECHOUCEK2, Karel BARTOS3, Martin GRILL4, PavelCELEDA5, and Vojtech KRMICEK6

1,2,3,4Department of Cybernetics and Center for Applied Cybernetics, Czech TechnicalUniversity in Prague5,6Institute of Computer Science, Masaryk University

ABSTRACTThe presented research aims to detect malicious traffic in high-speed networks by means ofcorrelated anomaly detection methods. In order to acquire the real-time traffic statistics inNetFlow format, we deploy transparent inline probes based on FPGA elements. They providetraffic statistics to the agent-based detection layer, where each agent uses a specific anomalydetection method to detect anomalies and describe the flows in its extended trust model. Theagents share the anomaly assessments of individual network flows that are used as an inputfor the agent’s trust models. The trustfulness values of individual flows from all agents arecombined to estimate their maliciousness. The estimate of trust is subsequently used to filterout the most significant events that are reported to network operators for further analysis.We argue that the use of trust model for integration of several anomaly detection methods andefficient representation of history data shall reduce the high rate of false positives (legitimatetraffic classified as malicious) which limits the effectiveness of current intrusion detectionsystems.

KEYWORDSIntrusion detection, network behavior analysis, multi-agent system, trust, anomaly detection

1 IntroductionIncreasing Internet traffic obliges the backbone oper-

ators and large end users to deploy high-speed networklinks to match the bandwidth demands. The increase ofbandwidth is apparent not only on backbone links, butthe consumer hosts are more and more connected withbandwidth capacity that was available only for enter-prise clients few years ago. However, besides all ben-eficial effects, this new high-bandwidth infrastructurepresents novel challenges in the domain of security androbustness, as the manual oversight of such high traf-fic volumes is nearly impossible and only the events ofextraordinary scale are typically reported [10].

Received September 13, 2007; Revised November 29, 2007; Accepted Decem-ber 11, 2007.1) [email protected], 2)[email protected],5)[email protected], 6)[email protected]

Our research aims to give the operators of these net-works a highly autonomous network intrusion detectionsystem (NIDS), which will use network behavior anal-ysis techniques as outlined in recent classification [17].Network behavior analysis systems are not based on thesignature matching, but use the network traffic statis-tics acquired in form of TCP/IP flows to identify rele-vant malicious traffic, such as vertical scanning (usedto determine the services offered by a host), horizontalscanning (used to map the network for on-line hosts,and used for worm and malware propagation [5]), de-nial of service attacks and other relevant events. Ourapproach does not aim to detect targeted compromiseof single hosts (this service being provided by IDS de-ployed on LANs), but concentrates on the events thatare significant on the network level, providing broaderperspective on current threats and allowing the opera-

DOI: 10.2201/NiiPi.2008.5.7

c©2008 National Instiute of Informatics

66 Progress in Informatics, No. 5, pp.65–74, (2008)

tors to address the most significant ones. Furthermore,the methods outlined in our system aim to detect the ac-tivity of the hosts in their networks that were taken overby an attacker (typically using zombie networks [5])and are used to stage further zombie recruiting, DoSattacks [10] or spam propagation.

Our work is based on the analysis of the networktraffic statistics. We aggregate the network data tocapture the information about network flows, unidirec-tional components of TCP connections (or UDP, ICMPequivalent) identified by shared source and destinationaddresses and ports, together with the protocol, and de-limited by the time frame used for data acquisition (seeSection 3.1). This information provides no hint aboutthe content of the transmitted data, but by detecting theanomalies in the list of flows acquired over the moni-toring period, we can detect irregularities and possibleattacks.

2 Related workIn order to detect an attack from the flow informa-

tion on the backbone level, especially without any feed-back from the affected hosts, it is necessary to ana-lyze the behavioral patterns in the traffic data, com-pare them with normal behavior and conclude whetherthe observed anomalies are caused by attacks, or aremere false alarms. This approach to Network Intru-sion Detection, typically based on the flow informa-tion captured by network flow probes is currently animportant field of research into Anomaly Based Intru-sion Detection, or more specifically Network BehaviorAnalysis. Numerous existing systems, based on traf-fic volume analysis modeled by Principal ComponentAnalysis (PCA) methods [7], models of entropy of IPheader fields for relevant subsets of traffic [9], [22], orjust count of the flows corresponding to the selectedcriteria [6] offer each a particular valid perspective onnetwork traffic.

The MINDS system [6] represents the flow by basicNetFlow aggregation features (srcIP, srcPrt, dstIP, dst-Prt, protocol) and complements them by the number ofthe flows from the same srcIP, to the same dstIP andtheir combinations with dstPrt and srcPrt respectively.These properties are assessed both in time and numberof connections defined windows, to account for slowscanning. The anomalous traffic is detected by meansof clustering, using local outlier factor, and the operatorcan use dedicated system modules to support him whencreating the rules describing identified attacks, so thatthey can be applied to future traffic.

The system proposed by Xu et al. [22] for traffic anal-ysis on backbone links also uses the NetFlow basedidentity 5-tuple. The context of the single connectionis defined by the normalized entropy of srcPrt, dstPrt

and dstIP dimensions of the set of all connections fromthe given host in the current time frame. These threedimensions define a feature space, where the methodapplies predefined rules to separate legitimate and ma-licious traffic.

Two methods adopted from the work of Lakhina etal. [8], [9] share several common properties: they wereoriginally designed for detection of anomalous origin-destination flows, defined as a sum of all flows arrivingfrom one network and leaving to another network. Inorder to identify the anomalous behavior, both variantsuse Principal Component Analysis (PCA) to model thenormal traffic, using the past flow sets to predict traf-fic volumes [8]. Differences between predicted and ob-served values are then used to identify the anomalies.

In the alternative approach [9], the PCA method isused to model the normal and residual entropy of thedestination IP addresses and source and destinationports of all flows originating from the same srcIP. Sub-tracting the modeled values from the observed data sub-tracts normal traffic entropies and allows us to concen-trate on anomalous residual entropy. Technically, theapproach is analogous to the previous method, the onlydifference is in the fact that we model three parametersinstead of one, and that these parameters are based onentropies, rather than volumes.

Besides the above mentioned sample of backboneanomaly detection mechanisms, there are numerous re-search and commercial systems designed to protect lo-cal networks. A typical representative of recently de-veloped system is a SABER [18], which addresses notonly threat detection, but attempts to actively protectthe system by automatically generated patches. Techni-cal perspective on many existing IDS systems, includ-ing SNORT [20] and other signature matching tech-niques that detect intrusions by detecting patterns spe-cific to known attacks in network traffic, can be foundin [13]. A good, even if slightly outdated review ofclassic research and systems in the domain is providedby [2].

3 ArchitectureIn our approach, we have decided not to develop a

novel detection method, but rather to integrate severalmethods [6], [8], [9], [22] with an extended trust mod-els of a specialized agent. This combination allows usto correlate the results of the used methods and to com-bine them to improve their effectiveness. Most anomalydetection methods today are not fit for commercial de-ployment due to the high ratio of misclassified traffic.While their current level of performance is a valid sci-entific achievement, the costs associated with supervi-sion of such systems are prohibitive for most organi-zations. Therefore, our main research goal is to com-

CAMNEP: An intrusion detection system for high-speed networks 67

Fig. 1 System architecture overview.

bine the efficient low-level methods for traffic obser-vation, with multi-agent detection process to detect theattacks with comparatively lower error rate, and to pro-vide the operator with efficient incident analysis layer.This layer supports operator’s decisions about detectedanomalies by providing additional information from re-lated data sources. The layer is also responsible forvisualization of the anomalies and the detection layerstatus.

The architecture consists of several layers with vary-ing requirements on on-line processing characteristics,level of reasoning and responsiveness. While the low-level layers need to be optimized to match the highwire-speed during the network traffic acquisition andpreprocessing, the higher layers use the preprocesseddata to infer the conclusions regarding the degree ofanomaly and consecutively also the maliciousness ofthe particular flow or a group of flows. Therefore, whilethe computation in the higher layers must be still rea-sonably efficient, the preprocessing by the lower layersallows us to deploy more sophisticated algorithms. Thesystem can be split into these layers, as shown in Fig. 1.

Traffic acquisition and preprocessing layerThe components in this layer acquire the data from

the network using the hardware accelerated NetFlowprobes [3] and perform their preprocessing. This ap-proach provides the real-time overview of all active uni-directional connections on the observed link. In orderto speed-up the analysis of the data, the preprocess-ing layer aggregates meaningful global and per-flow (orgroup of) characteristics and statistics.

Cooperative threat detection layerThis layer principally consists of specialized, hetero-

geneous agents that seek to identify the anomalies inthe preprocessed traffic data by means of their extendedtrust models [16]. Their collective decision regardingthe degree of maliciousness of a flow with certain char-acteristics use a reputation mechanism. The agents runinside the AGLOBE agent platform [19] and use itsadvanced features like agent migration and cloning toadapt the system to the traffic and relevant threats.

Operator and analyst interface layerThis layer is responsible for interaction with the net-

work operator. The main component is an intelligentvisualization agent that helps the operator to analyzethe output of the detection layer, by putting the pro-cessed anomaly information in context of other relevantinformation. When the detection layer detects suspi-cious behavior on the network, it is reported to visu-alization. The visualization agent then opens the newcase and retrieves relevant information from availabledata sources. The network operator can explore andevaluate the reported case subsequently. Another partof this layer is a set of lightweight, specialized visual-ization agents that allows the operator to follow onlythe selected characteristics of the system.

3.1 Traffic acquisition and preprocessingThe traffic acquisition and preprocessing layer is re-

sponsible for network traffic acquisition, data prepro-cessing and distribution to upper system layers. It onlyuses the flow characteristics based on information frompacket’s headers.

In general, each flow is a set of packets defined bycommon source and destination IP address, port num-bers and protocol. Flows are thus unidirectional andall their packets travel in the same direction. For theflow monitoring we use NetFlow protocol developed byCisco Systems [4].

The amount of traffic in nowadays high-speed net-works increases continuously and traffic characteristicschange heavily in time (network throughput fluctua-tion due to time of day, server backups, DoS attacks,scanning attacks, etc.). Performance of network probesmust be independent of such states and behave reliablyin all possible cases. The quality of provided data sig-nificantly effects the upper layers and chances to detecttraffic anomalies.

Therefore we use hardware accelerated NetFlowprobes called FlowMon. The FlowMon probe is a pas-sive network monitoring device based on the COMBOhardware [3], which provides high performance and ac-curacy. The probe handles 1 Gb/s traffic at line rate inboth directions and exports acquired NetFlow data todifferent collectors.

The collector servers store incoming packets withNetFlow data from FlowMon probes into its internaldatabase. Each collector server provides interface tographical and text representation of raw network traffic,simple flow filtration, aggregation and statistics evalu-ation, using source and destination IP addresses, portsand protocol.

To process acquired IP flows by upper system layersthe preprocessing must be performed on several levelsand in different manners. Packets can be sampled (ran-


dom, deterministic or adaptive sampling) on input andthe sampling information is added to NetFlow data. Onthe collector side several statistics (average traffic val-ues, entropy of flows) are computed. The collectorsprovide traffic statistics via tasd (Traffic AcquisitionServer Interface Daemon) interface to the AGLOBEagent platform.

Even after probes deployment in monitored network,the probes can be reprogrammed to acquire new traf-fic characteristics. The system is fully reconfigurableand the probes can adapt their features and behavior toreflect the changes in the agent layer.

3.2 Cooperative threat detectionThe goal of the cooperative threat detection layer is

to provide the assessment of maliciousness of the indi-vidual flows in each flow set observed by the system.To achieve this goal, we use the trust modeling tech-niques, and extend them to cover the domain-specificneeds.

Each detection agent contains one of the anomalydetection methods (discussed in Section 2 and detailedin [14]), coupled with an extended trust model definedin [16]:

• MINDS agent [6], which reasons about the num-ber of flows from and towards the hosts in the net-work, and detects the discrepancies between thepast and current traffic,

• Xu agent [22], which reasons about the traffic fromindividual hosts using the normalized entropiesand rules,

• Lakhina Entropy [9] agent, which builds a modelthat predicts the entropy of traffic features fromindividual hosts and identifies anomalies as differ-ences between predicted and real value, and

• Lakhina Volume agent [8], which applies the samemethod to traffic volumes.

All agents, regardless of their type, process the datareceived from the acquisition layer in three distinctstages (see Fig. 2):

• anomaly detection,• trust update,• collective trust conclusion.

Before the description of each these three stages, wewill define the terms and techniques used in our ap-proach. Trust of x in y is defined by Marsh as “x expectsthat y will behave according to x best interests, and willnot attempt to harm x” [12]. In the network security do-main, low trustfulness of the flow means that the flowis considered as a part of an attack. Trustfulness is de-termined in the [0, 1] interval, where 0 corresponds to

Fig. 2 Agent layer operation.

complete distrust and 1 to complete trust. The identityof each flow is defined by the features we can observedirectly on the flow: srcIP, dstIP, srcPrt, dstPrt, pro-tocol, number of bytes and packets. If two flows ina data set share the same values of these parameters,they are assumed to be identical. The context of eachflow is defined by the features that are observed on theother flows in the same data set, such as the numberof similar flows from the same srcIP, or entropy of thedstPrt of all requests from the same host as the evalu-ated flow. While the agents in our system use the samerepresentation of the identity, the context is defined bythe features used by their respective anomaly detectionmethods to draw the conclusions regarding the anomalyof the flow. Identity and context are used to define thefeature space, a metric space on which the trust modelof each agent operates [16]. The metrics of the spacedescribes the similarity between the identities and con-texts of the flows, and is specific to each agent.

Anomaly detectionDuring the anomaly detection stage, the agents use

the embedded anomaly detection method to determinethe anomaly of each flow as a value in the [0, 1] inter-val, where 1 represents the maximal anomaly, and zerono anomaly at all. The anomaly values are shared withother detection agents, and used as an input in the sec-ond phase of the processing.

Trust updateDuring the trust update, the agents integrate the

anomaly values determined for individual flows duringthe first phase into their trust models. As the reasoningabout the trustfulness of each individual flow is bothcomputationally infeasible and unpractical (the flows


are single shot events by definition), the model holdsthe trustfulness of significant flow samples (e.g. cen-troids of (fuzzy) clusters) in the identity-context space,and the anomaly of each flow is used to update the trust-fulness of centroids in its vicinity. The weight usedfor the update of the centroid’s trustfulness with theanomaly values provided for the flow decreases withdistance from the centroid. Therefore, as each agentuses a distinct distance function, each agent has a differ-ent insight into the problem — the flows are clusteredaccording to the different criteria, and the cross corre-lation implemented by sharing of the anomaly valuesused to update the trustfulness helps to eliminate ran-dom anomalies.

Collective trust estimationIn the last stage of processing, each agent determines

the trustfulness of each flow (with an optional normal-ization step), all agents provide their trustfulness as-sessment (conceptually a reputation opinion) to the ag-gregation agents and the visualization agents, and theaggregated values can then be used for traffic filtering.

In order to be successful, the trustfulness aggregatedby the system should be as close as possible to the ma-liciousness of the flow. When we reason about the ma-licious and untrusted flows as sets (they are actuallyfuzzy sets), we wish them to overlap as much as pos-sible. We can define the common misclassificationserrors using the trustfulness and maliciousness of theflow. The flows that are malicious and trusted are de-noted as false negatives, and the flows that are untrustedbut legitimate are denoted false positives. Typically,when we tune the system to reduce one of these sets, thesize of the other increases. Intuitively, it may seem thatwe may be ready to ignore higher rate of false positives,rather than false negatives. However, this is rarely thecase in the IDS systems deployed for operational use,as the legitimate traffic vastly outnumbers the attacksand even a low rate of false positives makes the systemunusable [1].

Performance of an isolated detection agent would besimilar to the performance of the anomaly detectionmethod it is based on. The application of trust mod-eling mechanism in the layer above the anomaly detec-tion allows the agents to eliminate probable false pos-itives identified as malicious only by a single method.We shall note that each agent represents the flows in itsfeature space, and that the context subspace definitiondepends on the anomaly detection algorithm applied bythe agent. The context of each flow depends on the (i)flow identity, (ii) characteristics of other (similar) flowsin the current observed flow set, and (iii) the currentstate of the agent’s anomaly detection model, whichis typically based on past flow sets. This implies two

defining features of our methodology, method integra-tion and history integration.

Method integrationProximity in the identity-context space of one of

the agents doesn’t imply the proximity in the identity-context space of another agent — as all agents use dif-ferent contexts to compare the traffic, the anomaliessignaled by a single agent will likely be noticed if theyare consistent with the other anomalies (provided by thesame agent or different agents) of similar flows, thatshare the centroids. The key aspect is that flow similar-ity is assessed by each trusting agent using its own crite-ria — when one of the agents signals an anomaly in theflow that falls into the centroid with mostly trusted traf-fic, this anomaly will be dispersed and will not manifestitself in the trustfulness evaluation. On the other hand,when the anomaly falls into/near the cluster with exist-ing lower trustfulness (or a new cluster), and is consis-tently reported by most anomaly providers, the agentwill return a low trustfulness for this flow. The Fig. 3,illustrates the concept — all attack flows are concen-trated next to a single centroid in agent’s trust model,and the situation is similar in the trust models of otheragents. The legitimate traffic is dispersed among otherclusters.

History integrationTrust model provides an efficient method for the in-

tegration of the past observations by aggregating thetrustfulness values from the history of anomalies of

Fig. 3 A peek into the trust model of a detection agent.Attack flows displayed as tree extremities attached tothe closest centroid of the trust model, shown as achrysanthemum-like structure on the right side of image. Inthe background, we can see other centroids, with the restof the more trusted traffic attached to them.


similar flows acquired in the previous data sets. Thisaggregation also helps to eliminate individual anomalypeaks specific to one flow set that are not consistentwith global assessment.

The overall impact of aggregations over history andover several anomaly detection algorithms aims toeliminate singular anomalies identified only occasion-ally by single anomaly detection method, and thus re-duce the number of false positives. On the other hand,some small-scale attacks may be missed by the system.We consider this as an acceptable trade-off, because thesystem is designed to provide warnings in the case oflarge-scale events, and not detailed protection of indi-vidual hosts.

It shall also be noted that our design helps to inte-grate various anomaly detection models. However, asthe computational performance of the system is oneof the main goals, each agent only uses the data thatis relevant to its anomaly detection (and consecutivelyflow representation features) method. This makes thedirect translation between the trust models of differenttypes of agents largely unfeasible — trustfulness is at-tached to centroids in their feature spaces, and there isno guarantee whatsoever regarding the compatibility ofthe metrics. Furthermore, direct translation is also diffi-cult for the agents sharing the same anomaly detectionmethod, but working with different data (acquired bydistinct probes). As the identity-context space metricsoften depends on the data observed in the past, directmerging of the models does not provide any guaranteesof consistence.

To compensate this problem, our architecture putsthe interactions at the processing stages where theagents share the same language, i.e. the identity of spe-cific network flows. When sharing the anomalies andtrustfulness (which may be normalized into degree oftrust [14]) values, these values are relevant to individ-ual network flows, and each agent is able to positionthem in its feature space.

In the last phase of evaluation, each agent shares thetrustfulness of flows with other agents. Agents thenuse a simple reputation mechanism to reach a collec-tive conclusion regarding the trustfulness of observedflows, allowing to filter the traffic by trustfulness foranalysis. Collectively accepted flows are then sent toanalyst interface layer for further filtering based on userpreferences, and possibly assisted analysis by analyst.

The decision whether a given flow is trusted or un-trusted depends on the typical degree of anomaly inthe observed network traffic. This parameter varieswidely with network type — the number of anomaliesis typically low on corporate or government networks,but is significantly higher on public Internet backbonelinks, or in the university settings. To avoid the prob-

lems with manual tuning of the system, we use a fuzzy-inference process integrated with the trust model to de-cide whether the given flow is malicious, by computingits inference with Low and High trust values. Thesevalues are determined similarly to the trustfulness ofindividual flow representations, but are represented astwo fuzzy intervals [15].

When we look at the problem from the ma-chine learning perspective [21], it is difficult to decidewhether the trust learning as implemented in our systemis supervised or unsupervised. It is clear that there is nodedicated supervisor entity, but each agent provides theinputs (i.e. anomalies) to the others, and their adapta-tion is mutual. On the other hand, these inputs are pro-vided by the anomaly detection methods of individualagents, and these can be in some cases considered assupervisors for the trustfulness learning.

4 System evaluation and performanceIn order to evaluate the effectiveness of our system,

we have deployed one probe at the main network gate-way of one of the university campuses, and we haveused the standard tools to attack the hosts inside the ob-served network, on the background of regular networktraffic. We present a selection of results that we haveobtained.

In the results, we want to validate several propertiesof the system. First and foremost, the system shall beable to identify the introduced attack as malicious. Thisrequires the agents to assign low trustfulness to the at-tack traffic, while assigning higher trustfulness to nor-mal traffic.

While this requirement is important, there are alsoother properties that we want to validate. As the trustmanagement system described in [14] and Section 3.2effectively performs fuzzy classification of flows to themore or less trusted classes defined by the centroids inthe identity-context space, we want the traffic relative tosingle attack to fall into the same class (or few classes).The trustfulness of these classes shall be low to com-ply with the previous requirement, but their low numberand proximity in the identity-context space shall allowclassification of attacks and signifies that the featuresused to define the space, the metrics using these fea-tures and the algorithm used for centroid creation arewell adjusted to detect this type of attack in the currentenvironment.

In the experimental results provided below, we illus-trate the capabilities of the system by showing the datafrom the actual attack we have launched against our ex-perimental platform. The attacks were performed usingthe nmap [11] tool, which is used by network admin-istrators and attackers alike to map the network and toidentify the services running on individual hosts. We


Fig. 4 Trustfulness of all flows as determined by theagents.

have performed several vertical scans of a single host,probing different ports for listening services using theSYN scan.

In Fig. 4, we can see the histogram of trustfulnessof the flows for 1024 ports SYN scan as determinedby detection agents [14] based on MINDS, Xu. et al.and the agent based on entropy prediction as introducedby Lakhina et al. (Lakhina Entropy). The trustfulnessis displayed on [0, 1] scale, 0 denotes complete dis-trust, while 1 corresponds to complete trust. Each ofthe agents maintains its trust model using the processesdescribed in Section 3.2, and we present the distribu-tion of the traffic on the trustfulness scale as assessedby each of them. We can note that the trustfulness ofmost traffic falls between 0.3 and 0.75 for the MINDSmethod, 0.4 and 0.6 for the Lakhina Entropy method,and 0.4 and 0.8 for the Xu method. We can clearlyobserve the ability of all agents to distinguish the un-trusted traffic on the left side of the distribution fromthe trusted traffic on the right side, but we can also seethat each agent provides a different shape of the his-togram.

In Fig. 5, we only present the traffic correspondingto our attack. We can distinguish the peaks observedon the first graph, and we can assert that the attack wehave generated was clearly identified by the method asan attack. For both Lakhina Entropy and Xu distri-butions, it even constitutes the lower boundary of thetrustfulness of the observed flows. When we analyzethe traffic contributing to the 0.3 peak on the left sideof the attack peak in the MINDS distribution, we canmostly distinguish peer-to-peer networks, heavy webuse (e.g. http proxies), and several clear examples offalse positives, mostly due to the irregularities in theactivity of observed servers. When we combine the re-

Fig. 5 Trustfulness of SYN scan attack flows as deter-mined by the agents. Note the different scale from Fig. 4.

Fig. 6 Aggregated trustfulness of all flows as averaged foreach flow from the values provided by the agents. Note theisolated peak on the left, which contains attack flows.

sults of the agents in the AND mode, the attack remainsthe only event (i.e. group of flows) which is classifiedwith trust lower than 0.5 by all three agents. This is eas-ily observed in Fig. 6, where we show the distributionof flows over the aggregated trustfulness. In this ob-servation, we can observe the attack flows forming anisolated peak on the left side of the distribution, around0.4, and that the rest of the traffic falls into the mainpart of the distribution. We can note the peak on theleft side of the main distribution — most of the trafficin this peak corresponds to the response to scan, and theassociated lower trustfulness is thus justified.

We also need to address the other criteria of quality.In the results shown above, the agents assign the flowscontributing to the attack to the neighborhood of veryfew centroids, showing that the trust models are set to


Fig. 7 Operator’s view of the attack through the GUI withapplied filtering.

an appropriate level of granularity [16] which makes atrade-off between precision and efficiency. In Fig. 3, wecan look into the trust model of the Lakhina Entropyagent, and we can see the attack flows attached to thesingle centroid (with low trustfulness) on the right frontside of the image, while the rest of the traffic is attachedto other centroids. The results for other agents are sim-ilar, and this consistency shows that the granularity ofthe trust model is well adapted to current network sta-tus.

While Fig. 3 provides an interesting insight into thereasoning process of the trusting agents, it is of limiteduse to network administrators that need more practicalrepresentation of data. Therefore, the system contains adedicated visualization agent which allows the adminis-trators to filter out the trusted traffic, as assessed by theagents, and to concentrate on the analysis of untrustedtraffic. In case of the attack presented above, the resultis shown in Fig. 7.

While the single attack attempt can illustrate thefunction of the system, it offers only a limited insightinto its effectiveness. Therefore, we present a graphwhich illustrates the effectiveness of the system in de-tecting the vertical scans of with different parameters,techniques (TCP CONNECT/SYN scan, UDP scan),speed and ordering. We will show the trustfulness at-tributed to the events by individual agents (includingthe aggregation) in function of the number of flows dur-ing the acquisition period. The Fig. 8 shows that the at-tacks with more than several hundreds flows (typicallycorresponding to number of ports scanned on the tar-get machine) are consistently discovered by all agents.The slower attacks, using lower number of flows (300and less) are more tricky. While the aggregated trustful-ness values are typically low, the higher average trust-fulness attributed to these attacks means that they canblend into the normal traffic.

The results we have presented show that the use oftrust models for the integration of several anomaly de-

Fig. 8 Trustfulness of various types of scanning attacksas estimated by the agents as a function of the numberof flows in the attack.

tection methods improves the results of network behav-ior analysis, and that this method can be efficiently in-tegrated with real traffic using efficient inline NetFlowprobes. While we have only presented the results rela-tive to identification of vertical scanning, the other re-sults show that our method is effective against other rel-evant threats, such as worms and other malware propa-gation, horizontal scanning, peer-to-peer networks andother irregularities. The evaluation of effectiveness ofour method against these threats is an important topicof our current and future work, together with improve-ments of the method itself.

5 Conclusions and future workThe research presented in this paper uses highly ab-

stract methods from the field of multi-agent trust mod-eling, and extends them to efficiently fulfill the needsof network operators: identification of significant mali-cious traffic events in the supervised network, withoutbeing bothered by comparatively high rate of false pos-itives.

To achieve this goal, we can not rely on agent tech-nologies alone. The data must be acquired from the net-work, preprocessed in the collector servers and trans-ferred to the detection layer for filtering. The detectionprocess is based on the combination of agent-specificanomaly detection techniques with extended trust mod-eling, that each agent uses to integrate the anomalyvalues from past observations, as well as anomaly as-sessments provided by other agents. This step reducessignificantly the number of false positives, as the non-systematic anomaly peaks are diluted between normaltraffic and anomaly values provided by other agents.

The results of the detection process performed inside


each agent are integrated by dedicated agents, using thereputation-like mechanism, and all trustfulness valuesare used during the traffic analysis by the operator. Thisanalysis is performed using the Mycroft agent, that notonly uses the trustfulness for shown traffic filtering, butalso facilitates the analysis by getting additional infor-mation about the traffic.

In our future work, we have several principal goals.First, the primary goal is to further increase the systemeffectiveness, to improve the ratio between false posi-tives and negatives. The performance of the system isalso crucial, and while we can still improve it by rigor-ous analysis and software engineering, we also seek al-ternative ways. The most promising is the autonomousadaptation — as the detection agents are self-contained,and the system is open, we will introduce the distributedadaptation mechanism that will select the best agents,well adapted to the current traffic status.

AcknowledgementThis material is based upon work supported by the

European Research Office of the US Army under Con-tract No. N62558-07-C-0001. Any opinions, findingsand conclusions or recommendations expressed in thismaterial are those of the author(s) and do not necessar-ily reflect the views of the European Research Officeof the US Army. Additional support provided by CzechMinistry of Education grants 1M0567 and 6840770038.Martin Rehak would also like to acknowledge the sup-port of the National Institute of Informatics in Tokyoduring his visit.

References[1] S. Axelsson, “The base-rate fallacy and the difficulty of

intrusion detection,” ACM Trans. Inf. Syst. Secur., vol.3, no. 3, pp.186–205, 2000.

[2] S. Axelsson, “Intrusion detection systems: A survey andtaxonomy,” Technical Report 99-15, Chalmers Univ.,March 2000.

[3] CESNET, z. s. p. o. Family of COMBO Cards. http://www.liberouter.org/hardware.php, 2007.

[4] Cisco Systems. Cisco IOS NetFlow. http://www.cisco.com/go/netflow, 2007.

[5] Evan Cooke, Farnam Jahanian, and Danny Mcpherson,“The Zombie Roundup: Understanding, Detecting, andDisrupting Botnets.” Workshop on Steps to ReducingUnwanted Traffic on the Internet (SRUTI), pp.39–44,June 2005.

[6] L. Ertoz, E. Eilertson, A. Lazarevic, P.-N. Tan, V.Kumar, J. Srivastava, and P. Dokas, “MINDS - Min-nesota Intrusion Detection System.” Next GenerationData Mining, MIT Press, 2004.

[7] A. Lakhina, M. Crovella, and C. Diot, “Characterizationof Network-Wide Anomalies in Traffic Flows.” ACM

SIGCOMM conference on Internet measurement IMC’04, pp.201–206, New York, NY, USA, ACM Press,2004.

[8] A. Lakhina, M. Crovella, and C. Diot, “DiagnosisNetwork-Wide Traffic Anomalies.” ACM SIGCOMM’04, pp.219–230, New York, NY, USA, ACM Press,2004.

[9] A. Lakhina, M. Crovella, and C. Diot, “Mining Anoma-lies using Traffic Feature Distributions.” ACM SIG-COMM, Philadelphia, PA, August 2005, pp.217–228,New York, NY, USA, ACM Press, 2005.

[10] M. Lesk, “The new front line: Estonia under cyberas-sault.” IEEE Security and Privacy, vol. 5, no. 4, pp.76–79, 2007.

[11] Gordon Lyon, Nmap. http://insecure.org/nmap/.

[12] S. Marsh, Formalising trust as a computational concept,1994.

[13] S. Northcutt and J. Novak, Network Intrusion Detection:An Analyst’s Handbook. Thousand Oaks, CA, USA,New Riders Publishing, 2002.

[14] M. Rehak, M. Pechoucek, K. Bartos, M. Grill, andP. Celeda, “Network intrusion detection by means ofcommunity of trusting agents.” IEEE/WIC/ACM Inter-national Conference on Intelligent Agent Technology(IAT 2007 Main Conference Proceedings) (IAT’07), LosAlamitos, CA, USA, 2007. IEEE Computer Society.

[15] M. Rehak, L. Foltyn, M. Pechoucek, and P. Benda.“Trust Model for Open Ubiquitous Agent Systems.” In-telligent Agent Technology, 2005 IEEE/WIC/ACM Inter-national Conference, number PR2416 in IEEE, 2005.

[16] M. Rehak and M. Pechoucek. “Trust modeling withcontext representation and generalized identities.” Co-operative Information Agents XI, number 4676 inLNAI/LNCS. Springer-Verlag, 2007.

[17] K. Scarfone and P. Mell, “Guide to intrusion detectionand prevention systems (idps).” Technical Report 800-94, NIST, US Dept. of Commerce, 2007.

[18] S. Sidiroglou and A. D. Keromytis. “Countering net-work worms through automatic patch generation.” IEEESecurity & Privacy, vol. 3, no. 6, pp.41–49, November/December 2005.

[19] D. Sislak, M. Rehak, M. Pechoucek, M. Rollo, and D.Pavlıcek, “A-globe: Agent development platform withinaccessibility and mobility support.” Software Agent-Based Applications, Platforms and Development Kits,pp.21–46, Berlin, Birkhauser Verlag, 2005.

[20] Sourcefire, Inc. SNORT – Intrusion Prevention System.http://www.snort.org/, 2007.

[21] J. Tozicka, M. Rovatsos, and M. Pechoucek, “A frame-work for agent-based distributed machine learning anddata mining.” Autonomous Agents and Multi-Agent Sys-tems (AAMAS 2007), pp.666–673, New York, NY, ACMPress, 2007.


[22] K. Xu, Z.-L. Zhang, and S. Bhattacharrya, “ReducingUnwanted Traffic in a Backbone Network.” USENIXWorkshop on Steps to Reduce Unwanted Traffic in theInternet (SRUTI), Boston, MA, July 2005.

Martin REHAKMartin REHAK works as a re-searcher at the Agent TechnologyGroup, Gerstner Laboratory, CzechTechnical University. He holds engi-neering degree from Ecole CentraleParis. His current research interests

are centered on trust, distributed system security, intru-sion detection and multi-agent task allocation.

Dr. Michal PECHOUCEKDr. Michal PECHOUCEK works asa reader in Artificial Intelligence andthe Head of Agent Technology Groupat the Department of Cybernetics,CTU. He holds degrees from FEE-CTU and University of Edinburgh

and completed his Ph.D. in AI at CTU in Prague.His research focuses on multi-agent systems, especiallytopics related to social knowledge, security, communi-cation inaccessibility, coalition formation, agent reflec-tion and multi-agent planning.

Pavel CELEDAPavel CELEDA works as a researcherat the Institute of Computer Scienceat the Masaryk University, Brno. Hehas a engineering degree in elec-trical engineering from the MilitaryAcademy Brno and Ph.D. in infor-

matics from University of Defence. His current re-search areas include monitoring of high-speed net-works and development of network probes.

Vojtech KRMICEKVojtech KRMICEK is a Ph.D. stu-dent at the Faculty of Informatics,Masaryk University, Brno, where hegot his master degree. His current re-search activities are focused on high-speed networks and network security.

He is a member of the software group in Liberouterproject, developing high-speed network monitoring de-vices.

Research Paper CAMNEP: An intrusion detection system for ...

Documents

Transcript of Research Paper CAMNEP: An intrusion detection system for ...