FPGA-based SoC for Real-Time Network Intrusion Detection...

FPGA-based SoC for Real-Time Network IntrusionDetection using Counting Bloom Filters

Jared Harwayne-Gidansky Deian Stefan and Ishaan DalalThe Center for Signal Processing Communications and Computer Engineering Research

The Cooper Union for the Advancement of Science and Art51 Astor Place Room 406B New York NY 10003 USA

harway stefan ishaancooperedu

AbstractmdashComputers face an ever increasing number ofthreats from hackers viruses and other malware effectiveNetwork Intrusion Detection (NID) before a threat affects end-user machines is critical for both financial and national securityAs the number of threats and network speeds increase (over 1gigabitsec) users of conventional software based NID methodsmust choose between protection or higher data rates

To address this shortcoming we have designed a hardware-based NID system-on-a-chip using data structures called Count-ing Bloom Filters (CBFs) Our design has extremely highthroughput (up to 33 gigabitssec) and can successfully detectand mitigate known threats and is to our knowledge the onlyknown CBF based NID system-on-a-chip to be implemented ona Virtex 4 FPGA

In this project we present the first optimized Counting BloomFilter based Network Intrusion Detection FPGA SoC (system-on-chip) implemented on a Virtex 4 FPGA our design is scalablethrough further parallelization and to our knowledge is one ofthe highest throughput NID systems in existence

Index TermsmdashData structures Computer network securityField programmable gate arrays

I INTRODUCTION

Viruses worms and hacker attacks on networks cost billionsof dollars (Klez virus $9 billion [1]) and affect hundreds ofthousands of users (MSBlast worm over 350000 hosts) [2]Additionally recent high profile attacks on our national secu-rity infrastructuremdashsuch as the infiltration of DoD networksby Chinese hackers [3]mdashreveal that defense against networkintrusion is now a matter of not just financial but also nationalsecurity

It is incorrect to assume that each user or individual machineon a network is secure [4] This is due to the large number ofmachines that need to be secured each of which requires timetechnical expertise and constant vigilance to ensure protectionagainst the latest threats To address these short comingsorganizations are moving toward Network Intrusion Detection(NID) preemptive detection of hacking attacks worms andother threats present in data when it enters the network iebefore it can reach the userrsquos machine

Network Intrusion Detection involves looking at patternsin network data that match known signatures stored in anexisting threat-database Currently NID is performed using

The design and results presented here were part of an undergraduate projectadvised by I Dalal at the Cooper Union

dedicated devices from manufacturers such as Cisco Thesedevices are essentially full-blown computers using softwaresolutions such as pattern matching with hash tables to performNID [5] However as both network data rates and the numberof potential threats increase the ability to effectively scan net-work data in real time using such devices becomes impracticalFor example while Gigabit (1000 megabitssec) networks areincreasing in popularity a typical Cisco NID device [5] hasa maximum throughput of only 150 megabitssec

As a result there has been a move toward custom hardwareimplementations of network intrusion detection which canhave significantly higher throughput [2] [6] Rather than fab-ricate custom ICs Field-Programmable Gate Arrays (FPGAs)are used for this purpose FPGAs are reconfigurable chipsthat contain programmable logic and can perform multiplecomplex operations in parallel and have become the dominantplatform for hardwarendashbased NID Such NID devices canoffer real-time protection against a wide array of threats whileachieving throughputs of over 1 gigabitsec [7] To ensurethe highest possible throughput we used Bloom filters anefficient data structure for hardware-based pattern matchingBloom Filters store a compact randomized representation ofthe threat-database allowing for a number of advantages overtraditional hash tables which have a large amount of overheadand require enough memory to store the full threat-databaseThe basic Bloom Filter can be improved into a CountingBloom Filter which allows for even more flexibility

In this project we present the first optimized counting-bloom-filter-based Network Intrusion Detection ldquosystem-on-chiprdquo (ie on a single FPGA chip) implemented on a Virtex4 FPGA our design is scalable through further parallelizationand to our knowledge is one of the highest throughput NIDsystems in existence

We first present the probabilistic mathematics that are thebasis of Bloom Filters and the evolution of Bloom Filtersinto Counting Bloom Filters We then discuss the designprocess and computer architectural challenges in hardwareimplementations of Bloom Filters Finally we use hardwareCounting Bloom Filters to set up a Network Intrusion De-tection system test it with real-world threats and presentperformance benchmarks

Outline

The outline of our paper is as followsbull Overview of Bloom Filters and Hash Functions

The structure and mathematical background of BloomFilters followed by performance analysis bullChoosing ahardware-efficient hash function for Bloom Filters

bull Counting Bloom Filters (CBFs) and overview ofFPGAs The limitations of Bloom Filters and theirevolution into CBFs bullDiscussion of trade-offs in CBFdesign bullA brief introduction to FPGAs and their suit-ability for Bloom Filter implementation

bull Hardware Implementations Three Bloom Filters andthree Counting Bloom Filters running at up to 33 giga-bitssec bullDesign results and conformance to theoreticalpredictions

bull Network Intrusion Detection using Counting BloomFilters Discussion implementation and analysis of aninnovative one-chip system for real time full-speed (giga-bit ethernet) detection of network threats using hardwareCounting Bloom Filters

bull Conclusion Review of presented material summary ofresults and discussion of future work

II OVERVIEW OF BLOOM FILTERS AND HASH FUNCTIONS

A Bloom Filter is a data structure used to determine if agiven piece of data belongs to a set [8] [9] To performthis ldquopattern-matchingrdquo Bloom Filters do not store the fullset but only a much smaller randomized representation thismakes them much more efficient than conventional hash tables[10]

A Definition

The goal of the Bloom Filter is to determine if a givenpiece of data (eg network packet) belongs to a predefined set(eg threat-database) Bloom Filters consist of two key partsan array of bits (known as the Bloom array and initialized toZEROs) and one or more hash functions The Bloom array hasa length of m-bits and a total of k hash functions First theBloom Filter must be programmed with a predefined set whichconsists of n elements This is accomplished by hashing eachelement of the set through the k hash functions the output ofthe hash functions are used as indices (addresses) for the m-bitarray and the bits at the corresponding addresses are lsquosetrsquo (iechanged to ONE)

After programming to check if a piece of data is a memberof the set (known as querying) that data is hashed If all thecorresponding bits of the Bloom array are set (ONE) then thatpiece of data is almost certainly part of the predefined setif any of the corresponding bits are not set (ZERO) the datais definitely not a part of the set (ie for NID there is nothreat) The check thus consists of performing an AND logicaloperation on the corresponding bits and checking if the outputis ZERO or ONE

Thus a Bloom Filter can definitely tell if a piece of datadoes not belong to the set (ie is not a threat) but there isa very small chance of a false positive data is identified as

H()U V

D

C

B

A

1

2

5

3

6

4

(a) An ideal hash functionH no collisions

1

2

5

3

6

4

G()U V

D

C

B

A

(b) A non-ideal hash func-tion G note the collision

Fig 1 Ideal and non-ideal hash functions mapping set U to set V

belonging to the set (a threat) when it actually is not This falsepositive probability can be controlled and made arbitrarilysmall so is not usually an issue

B Choosing Hash Functions for Hardware Bloom Filters

The Bloom Filterrsquos accuracy depends critically on the choiceof hash function [9] A hash function is essentially a one-way mapping between two sets For two sets U and V anideal (or perfectly random) hash function uniformly maps eachelement in U to a unique element in V (Fig 1a) Real-worldhash functions are rarely ideal and can have collisions ie themapping is no longer unique and more than one element inU may map to the same element in V (Fig 1b) Expressedmathematically for elements xy isin U x 6= y an ideal hashfunction H(middot) will have no collisions ie H(x) 6= H(y) on theother hand for a non-ideal hash function G(middot) collisions canoccur (ie G(x) = G(y))

Collisions in a Bloom Filter lead to false positives anddecrease its effectiveness Hence hash functions for BloomFilters in hardware must not only be as close to ideal as possi-ble (ie have a low collision rate) but also have low hardwarecomplexity (which allows for high throughput) Additionallysince the length of the Bloom array varies depending on theapplication the hash function should have variable-length out-puts Given these requirements we used the H3 hash functionsa member of the Universal Class of hash functions Thesehash functions are low-complexity randomized algorithms thatattempt to distribute the data inputs along the set of hashoutputs as evenly as possible with the appropriate choiceof random numbers they have low collision rates and can beconfigured for arbitrary length outputs [11] H3 hash functionsare an excellent choice for Bloom Filters and have recentlybeen successfully used for this purpose [7] [8]

C H3 Hash Functions

Given its desirable properties (low-complexity and lowcollision rate) we decided to use the H3 hash functionH3 hash functions can be computed with only exclusive-OR(XOR) operations making them very efficient for hardwareimplementation Let us illustrate H3 hashes with an exampleIf the data inputs to the hash function are say 8 bits and thehash outputs must be 4 bits we begin by choosing a matrix D

0 1 0 1 1 0 0 1

h1 h2 hk-1 hk

y

0

AND

1 1 1 1

h1 h2 hk-1 hk

xset elementhash functions

Bloom array

(a) Programming the filter (b) Checking membership

Fig 2 (a) Programming a Bloom Filter (b) Querying a Bloom Filter thedata being checked is not a member of the set (AND=0)

of eight 4-bit random numbers d1 d8 Then an elementx = 00011010 is hashed as

D =

d1d2d3d4d5d6d7d8

=

11000001010011101001011011011111

there4 h(x) = h(00011010)= x middotD= 0 middotd1oplus0 middotd2oplus0 middotd3

oplus1 middotd4oplus1 middotd5oplus0 middotd6

oplus1 middotd7oplus0 middotd8

= 1110oplus1001oplus1101

= 1010

where middot (AND) and oplus (XOR) are logical operations The resulth(x) is used to address the bit array of the Bloom Filter this isdemonstrated in Fig 2 By using different random matrices asmany different hash functions as necessary can be generated Itis important to note that H3 hash functions behave like an idealhash function when used with real-world data This is becausereal-world data can be treated as random for the purposes ofdetermining hash function performance [12]

D Formal definition of Bloom Filters

Mathematically a Bloom Filter is defined as a compactrepresentation of a set S = x1x2 xn of n elements storedin an array of m bits (all of which are initially ZERO) the khash functions h1 hk are independent and have a rangeof 1 m (ie each hash function maps uniformly to arandom number from 1 to m)

1) Programming the Bloom Filter (Fig 2a) The set S mustfirst be programmed in the Bloom Filter (eg S can be acertain set of known threats) For all x isin S each x is hashedby the k hash functions hi (1 le i le k) and the bits in theBloom array corresponding to the outputs hi(x) are set to ONE

as shown in Fig 2a2) Querying the Bloom Filter (Fig 2b) After the Bloom

Filter is programmed it is put into operation To check if anitem y is a member of S (eg for intrusion detection does apacket contain a virus signature y) you must hash it with allk hashes and then check to see if all the bits in the Bloomarray corresponding to hi(y) are set to ONE (Fig 2b) If theyare not ONE then y is definitely not a member of S but if theyare all ONEs then we assume y is a member of S althoughit may not be with a very small finite probability (ie a falsepositive) [9]

0110

1010

1110

0001

h1 h2 hk-1 hk


counter array

+_ +_ +_ +_

Fig 3 A Counting Bloom Filter with 4-bit updown counters

E False Positive Probability

A false positive occurs when an element that is not in theset is hashed and all the corresponding k bits in the Bloomarray turn out to be ONE the probability of this occurring is

n middot km

= ln(2) (1)

where k is the optimal number of hash functions for a givenm-bit Bloom array that will hold n entries

A derivation of this can be found in the appendix

III COUNTING BLOOM FILTERS AND FPGA OVERVIEW

A major limitation of Bloom Filters is that once an elementhas been programmed into the array it cannot be deletedwithout erasing and reprogramming the filter from scratchThis is because there is no way to reset the bits correspondingto one element to ZERO and ensure that none of those bitswere needed by another element still in the Bloom Filter Toaddress this deficiency Bloom Filters have been adapted intoCounting Bloom Filters (CBFs)

A CBF is identical to a standard Bloom Filter in concept butdiffers in implementation the Bloom array for a CBF consistsof counters and not individual bits (Fig 3) By using countersfor a CBF you can add and remove items from the CBFInstead of setting a bit when programming it you increment(+1) the corresponding counters when deleting an item youdecrement (minus1) the corresponding counters To query the CBFfor the existence of some piece of data you check if all thecorresponding counters in the Bloom array are non-zero (likechecking if all corresponding bits are set to ONEs in a standardBloom Filter)

A Designing Counting Bloom Filters

Since each counter in the CBF array has a limited size somenew challenges are introduced such as what to do if a counteroverflows because too many elements are added Empiricallya reasonable counter size that minimizes overflows for mostapplications is four bits [9]

Regardless of the counter size there is always the possibilitythat you reach the maximum value of a counter and anoverflow occurs in this situation it is best to leave the counterat its maximum value [9] While this could potentially causea false negative (which is impossible with standard BloomFilters) if the deletions from the CBF are more-or-less random(as is the case with real-world data) and the counter is of

LUT

BlockRAM

EmbeddedProcessor

(PowerPC)

LUT LUT LUT

LUT LUT LUT LUT

BlockRAM

(a) Block diagram of an FPGA (b) ML403 dev boardwith the Virtex-4 FX12FPGA (inset)

Fig 4 FPGA structure and our FPGA system-on-chip

a reasonable size (ie 4 bits) the average time until a falsenegative occurs is extremely large [9]

We used CBFs for our network intrusion detection as theyallow for dynamic re-programming (adddelete) of the filterThis flexibility is highly desirable to ensure that the filteris up to date eg if a new worm outbreak occurs (add)or if an attack is no longer a threat because the affectedsoftware was patched (delete) etc The CBFs drawbacks areoutweighed by their advantages the abundant resources on theFPGA (mitigating the memory requirement) and the very lowprobability of a false negative

B Overview of FPGA Architecture

The Field Programmable Gate Array is made up of 4-bitor 6-bit Look-Up Tables (LUTs) which can be programmedto compute any Boolean function of 4 or 6 inputs respec-tively Thousands of these LUTs make up an FPGA andare connected by reprogrammable interconnects allowing forimmense flexibility in implementing any digital function whichcan be optimized by performing many operations in parallelFig 4a shows the basic structure of a contemporary FPGAapart from LUTs the FPGA also contains blocks of fast staticrandom-access memories (block RAMs) and (optionally) anembedded microprocessor such as the PowerPC to performcontrol and data transfer functions The FPGA chip itself isusually integrated on a board that contains additional IO andstorage peripherals such as Ethernet controllers DDR RAMmodules and serial ports

The FPGA is an excellent platform for hosting a BloomFilter as it contains the fast RAM (ie Block RAMs) neededto minimize the time it takes access the bit array and its LUTsare ideal for performing multiple H3 hashes (XOR operations)in parallel It is due to this fact that a number of BloomFilter implementations on FPGAs exist for applications suchas spell checking medical imaging and network intrusiondetection [7] [8] [10] [13]

IV HARDWARE IMPLEMENTATIONS AND ANALYSIS

Before moving on to Network Intrusion Detection we firstdesigned optimized and implemented the necessary BloomFilters on the FPGA We have implemented three BloomFilters in both standard and Counting variants on the XilinxVirtex-4 FX12 FPGA hosted on the ML403 development

TABLE IRESULTS FOR BLOOM FILTER PROTOTYPES

(k=8 HASHES m=16384 BITS 4 BLOCK RAMS EACH)

Bloom FilterInput Size Freq

SlicesThroughput

(bytes) (MHz) (Gbps)

Regular 2 2051 127 328

4 2160 254 691

8 2006 529 128

Counting 2 2053 185 328

4 2039 319 652

8 2016 594 129

board (Fig 4b) The FPGA consists of 5472 slices (a slicecontains two LUTs) 36 block RAMs (of 18-kbits each) and anembedded PowerPC 405 processor The embedded processoreliminates the need for a separate computer to perform controlfunctions such as TCPIP network communication makingintelligent decisions in the NID threat detection process etcthis enabled us to make this a ldquosystem-on-a-chiprdquo

A Standard Bloom Filter Implementation

The Bloom array (m = 16384-bits) is stored in Block RAMon the FPGA To maximize throughput and minimize thecontrol and routing logic we give each hash function exclusiveaccess to the block RAM via a dedicated IO port We achievethis as follows since each block RAM has two IO ports (ieyou can perform two independent readswrites to differentaddresses per clock cycle) and we have k = 8 hashes wesplit the bit array across k

2 = 4 block RAMs that containm4 = 16384

4 = 4096 bits each While each hash function has to bemodified so its output falls within the range of its ldquoexclusiverdquoblock RAM this does not effect the false positive rate or theeffectiveness of the Bloom Filter [7] An AND gate is used toperform the final membership check

B Counting Bloom Filter Implementation

As described in section III going from a standard BloomFilter to a counting Bloom Filter design simply involvesmodifying the Bloom array so that each ldquobitrdquo is a 4-bitcounter instead We implemented our CBF by storing thecounter values in block RAM each block RAM (there are36 on the FPGA) has a capacity of 18-kbits The RAMwas used at a 4-bit depth and 4-bit adders were includedto incrementdecrement the RAM counters while comparatorswere used to check if the counters were lsquosetrsquo (ie its valuewas greater than zero) Using the LUTs as the ldquocountersrdquowas considered (as a counter is a 4-bit Boolean function) butrejected due to an insufficient number of LUTs on the FPGAand the large routing overhead which would have substantiallydecreased performance

CBFs are somewhat slower at programming than a standardBloom Filter while the standard Bloom Filter takes one clockcycle to change a bit the CBF takes two clock cycles tomodify a counter the original value must be read incremented(for an add) or decremented (for a delete) and then written

Bit Array

H1 H8hellip8-byte CBF

Bit Array

PreProcessing

PowerPCProcessor

DataIn

CheckedData Out


Bit Array

H1 Hkhellip2-byte CBF

Hash Table(RAM)

H1 H8

2-byte CBF

Alert to Destination

NID System on a Chip

Fig 5 A simplified block diagram of our NID design

back to update the counter The extra programming time isimmaterial since after the initial programming the CBF arrayonly needs occasional reprogramming (for updates) We notethat checking (reading) the CBF array which is by far themajor operation in an NID device only requires a single clockcycle thus using CBFs has negligible effect on throughputcompared to standard Bloom Filters

Our maximum throughput was approximately 33 Gbps acomplete summary of our results for the CBF prototype canbe found in Table I

V APPLICATION NETWORK INTRUSION DETECTION

As mentioned earlier networks face a large variety of threatsfrom hackers and malware that can cause damage in thebillions of dollars and breach national security Assumingthat end-users can protect their systems is both inaccurateand ineffective Detecting malicious data or events at thenetwork boundaries (Network Intrusion Detection) allows forpreemptive and broader protection from threats and creates asingle point to maintain network security (as opposed to atevery machine in the network) The NID device is installed atthe gateway to the network it is protecting

We have designed and implemented a Network IntrusionDetection (NID) device using Counting Bloom Filters to detectand mitigate threats before they enter a network

Fig 5 shows the structure of our NID device The inputdatamdashEthernet packets (frames)mdashis fed to the preproces-sor which extracts the IP packet stripping the headers andforwarding the host information to the PowerPC and theTCPUDP payload to the Counting Bloom Filters The filterstake 2- 4- and 8-byte data inputs respectively this is becausethe threat patterns (such as ILOVEYOU for the Love Bug worm)can be of variable size the preprocessor must be used tokeep track of patterns longer than 8-bytesmdashwe have notimplemented this capability yet

A flow chart showing how our NID device operates is shownin Fig 7 After the CBFs an intelligent processor is neededto analyze the output of the CBFs and decide if a threat isdetected Instead of offloading this to a separate computer ormicroprocessor (eg as is done in some commercial NIDsystems [5]) we decided to design the NID device as acomplete system-on-chip by using the embedded PowerPCprocessor Once the PowerPC detects a threat from the CBFoutputs it eliminates the possibility of a false positive by

128 256 512 1024 1536 20480

05

1

15

2

25

3

Signatures in the CBF

Per

cen

t o

f F

alse

Po

siti

ves

ExperimentalTheoretical

(a) 2-byte Bloom filter results

128 256 512 1024 1536 20480

05

1

15

2

25

3


Per

cen

t o

f F

alse

Po

siti

ves


(b) 4-byte Bloom filter results

128 256 512 1024 1536 20480

05

1

15

2

25

3


Per

cen

t o

f F

alse

Po

siti

ves


(c) 8-byte Bloom filter results

Fig 6 Experimental vs Theoretical False Positive Rates for (a) a 2-byteBloom filter (b) a 4-byte Bloom filter and (c) a 8-byte Bloom filter (m =16384-bits k = 8 hash functions and n = 128 2048 elements)

verifying if that piece of data in question is present in thesecondary hash table (stored in DDR RAM on the FPGAboard) Note that this hybrid ldquobloom filter + small secondaryhash tablerdquo approach is much faster and uses less memorythan a full hash-table approach

If the threat is confirmed the NID device drops thedata packet and sends an Internet Control Message Protocol(ICMP) datagram of type DESTINATION UNREACHABLE withexplanation code ldquoHost administratively prohibitedrdquo to thesource of the malicious packet Furthermore a UDP packetis sent to a logging facility where the system administratorcan further analyze the alerts The NID device can also takemore drastic steps such as IP blocking of knownpersistenthosts to avoid denial of service attacks on the NID deviceAdditionally since we use CBFsndashas opposed to standardBloom Filtersndashan administrator is able to modify the database(add or remove signatures) in real-time using the serial port onthe ML403 development board to rapidly respond to emergingthreats

NID System-on-a-Chip Testing and Performance

For our proof-of-concept NID implementation we chosea realistically-sized n = 2048 element data set containingsignatures for a variety of intrusions viruses trojans wormsdenial of service attacks brute-force unauthorized accessattempts and spam The signatures are based on a subset ofthe malicious payloads included in the industry-standard Snort(ver 24) rule-set [14]

Given n = 2048 with the additional constraint of a smallfalse positive probability (lt 25) to arrive at an optimalBloom array length of m = 16384 bits and k = 8 hash functions(this was done using eqns (5) and (6) found in the Appendix)Since the signatures vary in length each Bloom Filter acceptsa different-sized input 2- 4- or 8-bytes respectively

In evaluating the performance of the Bloom Filters weprogrammed each of the filters with an increasing number ofelements from n = 128 to 2048 based on the Snort rule-setand custom-crafted signatures testing the false positive rateon simulated traffic It is important to note that the custom-crafted signatures were used in cases where the number ofSnort payload signatures was not sufficient (ie lt n)

Based on real world data [15] and the testing methodologiespresented in [16] we simulated random traffic on the orderof asymp 70000 connections of which 258 were attacks (ieconnections whose payloads contained signatures which wereused to program the filters) The number of normal and attackconnections in a time interval were distributed according to aPoisson process resulting in normal and ldquoattackrdquo intervals inan attack interval the majority of the connections had payloadswith malicious signatures [16]

Fig 6 compares our Bloom Filterrsquos false positive ratewith the false positive rate of a theoretical Bloom Filter forsignatures of length 2-bytes (6a) 4-bytes (6b) and 8-bytes (6c)As is apparent our experimental false positive rates closelytrack the theoretical values

Ethernet packet (frame) enters the NID system

Pre-processor remove headers extract dataPre-processor remove headers extract data

Does any CBF (248-

byte) detect a threat

PowerPC is threat

a false positive

No action

forward

packet to

destination

No action

forward

packet to

destination

Drop packet terminate connection

NO THREAT

Send alert to admin

Next packet

YES

NO

YES

Fig 7 Flowchart of Network Intrusion Detection using CBFs on an FPGA

VI CONCLUSION

We presented a single-chip FPGA-based hardware NetworkIntrusion Detection (NID) system using Counting Bloom Fil-ters Our design identifies and can neutralize threats such ashackers and viruses at the network boundary before they canattack end-user computers As network data rates increasesuch real-time preemptive action may prove impractical forsoftware NID solutions

Counting Bloom Filters are efficient randomized data struc-tures that are much faster and use less memory than the hashtables usually used in software applications Consequentlyour NID device has a throughput of sim33 Gbpsmdashover anorder of magnitude higher than commercial software-basedNID systems

Future work includes increasing the number and variety ofthreat-signatures that the system can detect as well as fullscale testing on a live network We hope that by increasing theflexibility and speed of effective Network Intrusion Detectionwe can help secure computers against malicious attacksreduce associated financial losses and prevent the compromiseof national security

APPENDIXFALSE POSITIVE PROBABILITY DERIVATION

Here we derive the probability of getting a false positivewhen querying a Bloom Filter

Consider an m-bit array since the output of each hashfunction is uniformly and independently distributed over therange 1 m the probability that a bit is set (ie ONE)after a single hash of one element is 1

m The probability thata bit is not set (ie ZERO) is therefore (1minus 1

m )To program a Bloom Filter using k hash functions on a

set with n entries a total of n middot k hashing operations are

performed The probability p that a bit is not set after thefilter is completely programmed is therefore

p = Prob(bit is not set) =(

1minus 1m

)nmiddotk(2)

Assuming m is large (which is true in practice) p can beapproximated as

pasymp limmrarrinfin

(1minus 1

m

)nmiddotk=

[lim

mrarrinfin

(1+

1minusm

)minusm] nmiddotkminusm

(3)

where the expression inside the square brackets is thedefinition of the exponential (e) function Thus the probabilityp that a bit is not set is approximately

p = Prob(bit is not set)asymp eminusnmiddotk

m (4)

The complementary probability (pprime) that a bit is set to ONE

is simply pprime = (1minus p) A false positive occurs when all k bitsfor the entry being hashed are set to ONE thus the false positiveprobability p(FP) is

p(FP) = Prob(k bits are set)

= (pprime)k = (1minus p)k

asymp (1minus eminusnmiddotkm )k (5)

Given eq (5) the false positive probability p(FP) is mini-mized when

n middot km

= ln(2) (6)

which also gives the optimal value for one of the threevariables mnk if the other two are kept constant eg theoptimal number of hash functions k for a given m-bit Bloomarray that will hold n entries is k = m

n middot ln(2)

REFERENCES

[1] R Lemos ldquoCounting the cost of slammerrdquo CNet Feb 2003[2] J W Lockwood J Moscola M Kulig D Reddick and T Brooks

ldquoInternet worm and virus protection in dynamically reconfigurablehardwarerdquo in Proc of Mil and Aero Programmable Logic Device(MAPLD) Sep 2003

[3] J Swartz ldquoChinese hackers seek US accessrdquo USA Today Mar 2007[4] B Achoido and M Kessler ldquoWorm virus threat growsrdquo USA Today

Apr 2003[5] ldquoInstalling Cisco intrusion prvention system appliances and modules

51rdquo Cisco[6] M Attig and J Lockwood ldquoSIFT SNORT intrusion filter for TCPrdquo in

Proc 13th Symp on High Performance Interconnects Aug 2005 pp121ndash127

[7] S Dharmapurikar P Krishnamurthy T Sproull and J LockwoodldquoDeep packet inspection using parallel bloom filtersrdquo in Hot Intercon-nects Aug 2003 pp 44ndash51

[8] I L Dalal and F L Fontaine ldquoA reconfigurable FPGA-based 16-channelfront-end for MRIrdquo in 40th Asilomar Conf on Sigs Sys and Comps2006 OctNov 2006 pp 1860ndash1864

[9] A Broder and M Mitzenmacher ldquoNetwork applications of BloomFilters A surveyrdquo Internet Mathematics vol 1 no 4 pp 485ndash5092004

[10] H Song S Dharmapurikar J Turner and J Lockwood ldquoFast hash tablelookup using extended bloom filter an aid to network processingrdquo inSIGCOMM rsquo05 2005 pp 181ndash192

[11] M Ramakrishna E Fu and E Bahcekapili ldquoA performance study ofhashing functions for hardware applicationsrdquo in Intrsquol Conf Computingand Information 1994 pp pp 1621ndash1636

[12] M Mitzenmacher and S Vadhan ldquoWhy simple hash functions workexploiting the entropy in a data streamrdquo in Proc of the 19th annualACM-SIAM SODA 2008 pp 746ndash755

[13] M Attig S Dharmapurikar and J Lockwood ldquoImplementation re-sults of bloom filters for string matchingrdquo in 12th Symp on Field-Programmable Custom Computing Machines Apr 2004 pp 322ndash323

[14] M Roesch ldquoSNORT-lightweight intrusion detection for networksrdquo inLISA 1999 USENIX 13th Systems Administration Coinferecne Nov1999 pp 229ndash238

[15] R Lippmann R Cunningham D Fried I Graf K Kendall S Websterand M Zissman ldquoResults of the DARPA 1998 Offline Intrusion Detec-tion Evaluationrdquo in Recent Advances in Intrusion Detection vol 991999 pp 829ndash835

[16] J Cabrera B Ravichandran and R Mehra ldquoStatistical Traffic Modelingfor Network Intrusion Detectionrdquo in Proceedings of the 8th InternationalSymposium on Modeling Analysis and Simulation of Computer andTelecommunication Systems IEEE Computer Society Washington DCUSA 2000 pp 466ndash473

Jared Harwayne-Gidansky (Srsquo06ndashGSrsquo08) receivedthe BE degree in Electrical Engineering from theCooper Union for the Advancement of Science andArt He is currently a Graduate Electrical Engi-neering student at The Cooper Union a GraduateResearch Fellow with the SProCom2 research lab-oratory at Cooper Union and a reviewer for IEEEPotentials His interests include Control TheoryStochastic Processes Signals Processing Program-ming and Applied Mathematics

Deian Stefan (Srsquo06) will receive the BE degreein Electrical Engineering in 2009 from the CooperUnion for the Advancement of Science and ArtHe is currently an Undergraduate Research Fellowin the SProCom2 research laboratory at CooperUnion his interests include signal processing re-configurable systems cryptography and operatingsystems

Ishaan Dalal (Srsquo02ndashGSrsquo07) received the BE de-gree in Electrical Engineering from the CooperUnion for the Advancement of Science and Art in2006 Since then he has been a Graduate ResearchFellow with the SProCom2 research laboratory atCooper Union and an Adjunct Instructor of Elec-trical Engineering His interests include statisticalsignal processing information theory and system de-sign for communications He is currently pursuing adoctorate in Electrical Engineering at the Universityof Texas-Austin

Outline

The outline of our paper is as followsbull Overview of Bloom Filters and Hash Functions

The structure and mathematical background of BloomFilters followed by performance analysis bullChoosing ahardware-efficient hash function for Bloom Filters

bull Counting Bloom Filters (CBFs) and overview ofFPGAs The limitations of Bloom Filters and theirevolution into CBFs bullDiscussion of trade-offs in CBFdesign bullA brief introduction to FPGAs and their suit-ability for Bloom Filter implementation

bull Hardware Implementations Three Bloom Filters andthree Counting Bloom Filters running at up to 33 giga-bitssec bullDesign results and conformance to theoreticalpredictions

bull Network Intrusion Detection using Counting BloomFilters Discussion implementation and analysis of aninnovative one-chip system for real time full-speed (giga-bit ethernet) detection of network threats using hardwareCounting Bloom Filters

bull Conclusion Review of presented material summary ofresults and discussion of future work

II OVERVIEW OF BLOOM FILTERS AND HASH FUNCTIONS

A Bloom Filter is a data structure used to determine if agiven piece of data belongs to a set [8] [9] To performthis ldquopattern-matchingrdquo Bloom Filters do not store the fullset but only a much smaller randomized representation thismakes them much more efficient than conventional hash tables[10]

A Definition

The goal of the Bloom Filter is to determine if a givenpiece of data (eg network packet) belongs to a predefined set(eg threat-database) Bloom Filters consist of two key partsan array of bits (known as the Bloom array and initialized toZEROs) and one or more hash functions The Bloom array hasa length of m-bits and a total of k hash functions First theBloom Filter must be programmed with a predefined set whichconsists of n elements This is accomplished by hashing eachelement of the set through the k hash functions the output ofthe hash functions are used as indices (addresses) for the m-bitarray and the bits at the corresponding addresses are lsquosetrsquo (iechanged to ONE)

After programming to check if a piece of data is a memberof the set (known as querying) that data is hashed If all thecorresponding bits of the Bloom array are set (ONE) then thatpiece of data is almost certainly part of the predefined setif any of the corresponding bits are not set (ZERO) the datais definitely not a part of the set (ie for NID there is nothreat) The check thus consists of performing an AND logicaloperation on the corresponding bits and checking if the outputis ZERO or ONE

Thus a Bloom Filter can definitely tell if a piece of datadoes not belong to the set (ie is not a threat) but there isa very small chance of a false positive data is identified as

H()U V

D

C

B

A

1

2

5

3

6

4

(a) An ideal hash functionH no collisions

1

2

5

3

6

4

G()U V

D

C

B

A

(b) A non-ideal hash func-tion G note the collision

Fig 1 Ideal and non-ideal hash functions mapping set U to set V

belonging to the set (a threat) when it actually is not This falsepositive probability can be controlled and made arbitrarilysmall so is not usually an issue

B Choosing Hash Functions for Hardware Bloom Filters

The Bloom Filterrsquos accuracy depends critically on the choiceof hash function [9] A hash function is essentially a one-way mapping between two sets For two sets U and V anideal (or perfectly random) hash function uniformly maps eachelement in U to a unique element in V (Fig 1a) Real-worldhash functions are rarely ideal and can have collisions ie themapping is no longer unique and more than one element inU may map to the same element in V (Fig 1b) Expressedmathematically for elements xy isin U x 6= y an ideal hashfunction H(middot) will have no collisions ie H(x) 6= H(y) on theother hand for a non-ideal hash function G(middot) collisions canoccur (ie G(x) = G(y))

Collisions in a Bloom Filter lead to false positives anddecrease its effectiveness Hence hash functions for BloomFilters in hardware must not only be as close to ideal as possi-ble (ie have a low collision rate) but also have low hardwarecomplexity (which allows for high throughput) Additionallysince the length of the Bloom array varies depending on theapplication the hash function should have variable-length out-puts Given these requirements we used the H3 hash functionsa member of the Universal Class of hash functions Thesehash functions are low-complexity randomized algorithms thatattempt to distribute the data inputs along the set of hashoutputs as evenly as possible with the appropriate choiceof random numbers they have low collision rates and can beconfigured for arbitrary length outputs [11] H3 hash functionsare an excellent choice for Bloom Filters and have recentlybeen successfully used for this purpose [7] [8]

C H3 Hash Functions

Given its desirable properties (low-complexity and lowcollision rate) we decided to use the H3 hash functionH3 hash functions can be computed with only exclusive-OR(XOR) operations making them very efficient for hardwareimplementation Let us illustrate H3 hashes with an exampleIf the data inputs to the hash function are say 8 bits and thehash outputs must be 4 bits we begin by choosing a matrix D

0 1 0 1 1 0 0 1

h1 h2 hk-1 hk

y

0

AND

1 1 1 1

h1 h2 hk-1 hk


Bloom array




D =

d1d2d3d4d5d6d7d8

=

11000001010011101001011011011111





= 1010







0110

1010

1110

0001

h1 h2 hk-1 hk


counter array

+_ +_ +_ +_




n middot km

= ln(2) (1)









LUT

BlockRAM

EmbeddedProcessor

(PowerPC)

LUT LUT LUT

LUT LUT LUT LUT

BlockRAM













SlicesThroughput


Regular 2 2051 127 328

4 2160 254 691

8 2006 529 128

Counting 2 2053 185 328

4 2039 319 652

8 2016 594 129









Bit Array


Bit Array

PreProcessing

PowerPCProcessor

DataIn

CheckedData Out


Bit Array


Hash Table(RAM)

H1 H8

2-byte CBF











128 256 512 1024 1536 20480

05

1

15

2

25

3


Per

cen

t o

f F

alse

Po

siti

ves



128 256 512 1024 1536 20480

05

1

15

2

25

3


Per

cen

t o

f F

alse

Po

siti

ves



128 256 512 1024 1536 20480

05

1

15

2

25

3


Per

cen

t o

f F

alse

Po

siti

ves














Does any CBF (248-


PowerPC is threat

a false positive

No action

forward

packet to

destination

No action

forward

packet to

destination


NO THREAT

Send alert to admin

Next packet

YES

NO

YES


VI CONCLUSION












1minus 1m

)nmiddotk(2)



(1minus 1

m

)nmiddotk=

[lim

mrarrinfin

(1+

1minusm


(3)



m (4)







n middot km

= ln(2) (6)


n middot ln(2)

REFERENCES




















0 1 0 1 1 0 0 1

h1 h2 hk-1 hk

y

0

AND

1 1 1 1

h1 h2 hk-1 hk


Bloom array




D =

d1d2d3d4d5d6d7d8

=

11000001010011101001011011011111





= 1010







0110

1010

1110

0001

h1 h2 hk-1 hk


counter array

+_ +_ +_ +_




n middot km

= ln(2) (1)









LUT

BlockRAM

EmbeddedProcessor

(PowerPC)

LUT LUT LUT

LUT LUT LUT LUT

BlockRAM













SlicesThroughput


Regular 2 2051 127 328

4 2160 254 691

8 2006 529 128

Counting 2 2053 185 328

4 2039 319 652

8 2016 594 129









Bit Array


Bit Array

PreProcessing

PowerPCProcessor

DataIn

CheckedData Out


Bit Array


Hash Table(RAM)

H1 H8

2-byte CBF











128 256 512 1024 1536 20480

05

1

15

2

25

3


Per

cen

t o

f F

alse

Po

siti

ves



128 256 512 1024 1536 20480

05

1

15

2

25

3


Per

cen

t o

f F

alse

Po

siti

ves



128 256 512 1024 1536 20480

05

1

15

2

25

3


Per

cen

t o

f F

alse

Po

siti

ves














Does any CBF (248-


PowerPC is threat

a false positive

No action

forward

packet to

destination

No action

forward

packet to

destination


NO THREAT

Send alert to admin

Next packet

YES

NO

YES


VI CONCLUSION












1minus 1m

)nmiddotk(2)



(1minus 1

m

)nmiddotk=

[lim

mrarrinfin

(1+

1minusm


(3)



m (4)







n middot km

= ln(2) (6)


n middot ln(2)

REFERENCES




















LUT

BlockRAM

EmbeddedProcessor

(PowerPC)

LUT LUT LUT

LUT LUT LUT LUT

BlockRAM













SlicesThroughput


Regular 2 2051 127 328

4 2160 254 691

8 2006 529 128

Counting 2 2053 185 328

4 2039 319 652

8 2016 594 129









Bit Array


Bit Array

PreProcessing

PowerPCProcessor

DataIn

CheckedData Out


Bit Array


Hash Table(RAM)

H1 H8

2-byte CBF











128 256 512 1024 1536 20480

05

1

15

2

25

3


Per

cen

t o

f F

alse

Po

siti

ves



128 256 512 1024 1536 20480

05

1

15

2

25

3


Per

cen

t o

f F

alse

Po

siti

ves



128 256 512 1024 1536 20480

05

1

15

2

25

3


Per

cen

t o

f F

alse

Po

siti

ves














Does any CBF (248-


PowerPC is threat

a false positive

No action

forward

packet to

destination

No action

forward

packet to

destination


NO THREAT

Send alert to admin

Next packet

YES

NO

YES


VI CONCLUSION












1minus 1m

)nmiddotk(2)



(1minus 1

m

)nmiddotk=

[lim

mrarrinfin

(1+

1minusm


(3)



m (4)







n middot km

= ln(2) (6)


n middot ln(2)

REFERENCES




















Bit Array


Bit Array

PreProcessing

PowerPCProcessor

DataIn

CheckedData Out


Bit Array


Hash Table(RAM)

H1 H8

2-byte CBF











128 256 512 1024 1536 20480

05

1

15

2

25

3


Per

cen

t o

f F

alse

Po

siti

ves



128 256 512 1024 1536 20480

05

1

15

2

25

3


Per

cen

t o

f F

alse

Po

siti

ves



128 256 512 1024 1536 20480

05

1

15

2

25

3


Per

cen

t o

f F

alse

Po

siti

ves














Does any CBF (248-


PowerPC is threat

a false positive

No action

forward

packet to

destination

No action

forward

packet to

destination


NO THREAT

Send alert to admin

Next packet

YES

NO

YES


VI CONCLUSION












1minus 1m

)nmiddotk(2)



(1minus 1

m

)nmiddotk=

[lim

mrarrinfin

(1+

1minusm


(3)



m (4)







n middot km

= ln(2) (6)


n middot ln(2)

REFERENCES






























Does any CBF (248-


PowerPC is threat

a false positive

No action

forward

packet to

destination

No action

forward

packet to

destination


NO THREAT

Send alert to admin

Next packet

YES

NO

YES


VI CONCLUSION












1minus 1m

)nmiddotk(2)



(1minus 1

m

)nmiddotk=

[lim

mrarrinfin

(1+

1minusm


(3)



m (4)







n middot km

= ln(2) (6)


n middot ln(2)

REFERENCES






















1minus 1m

)nmiddotk(2)



(1minus 1

m

)nmiddotk=

[lim

mrarrinfin

(1+

1minusm


(3)



m (4)







n middot km

= ln(2) (6)


n middot ln(2)

REFERENCES




















FPGA-based SoC for Real-Time Network Intrusion Detection...

Documents

Transcript of FPGA-based SoC for Real-Time Network Intrusion Detection...