Monitoring, Diagnosing, and Securing the Internet 1 Yan Chen Department of Electrical Engineering and Computer Science Northwestern University Lab for Internet & Security Technology (LIST) 2 3 The Spread of Sapphire/Slammer Worms 4 Current Intrusion Detection Systems (IDS) Mostly host-based and not scalable to high- speed networks Slammer worm infected 75,000 machines in certain threshold Intrinsic to buffer overflow vulnerability and hard to evade Vulnerable buffer Protocol message Overflow! Vision of NetShield 16 17 Motivation Desired Features for Signature-based NIDS/NIPS Accuracy (especially for IPS) Speed Coverage: Large ruleset Regular Expression Vulnerability AccuracyRelative Poor Much Better SpeedGood?? MemoryOK?? CoverageGood?? Shield [sigcomm04] Focus of this work Cannot capture vulnerability condition well! 18 Research Challenges Background Use protocol semantics to express vulnerability Protocol state machine & predicates for each state Example: ver==1 && method==put && len(buf)>300 Challenges Matching thousands of vulnerability signatures simultaneously Sequential matching algorithmic parallel matching High speed parsing Applicability for large NIDS/NIPS rulesets 19 Outline Motivation Feasibility Study: a Measurement Approach Given a large NIDS/NIPS ruleset, what percentage of the rules can be improved with protocol semantic vulnerability signatures? High Speed Parsing High Speed Matching for Large Rulesets. Evaluation Conclusions 20 Measure Snort Rules Semi-manually classify the rules. 1.Group by CVE-ID 2.Manually look at each vulnerability Results 86.7% of rules can be improved by protocol semantic vulnerability signatures. Most of remaining rules (9.9%) are web DHTML and scripts related which are not suitable for signature based approach. On average 4.5 Snort rules are reduced to one vulnerability signature. For binary protocol the reduction ratio is much higher than that of text based ones. For netbios.rules the ratio is 67.6. 21 Outline Motivation Feasibility Study: a Measurement Approach High Speed Parsing High Speed Matching for Large Rulesets. Evaluation Conclusions 22 Observations array PDU PDU parse tree Leaf nodes are integers or strings Vulnerability signatures mostly based on leaf nodes Observation 1: Only need to parse the fields related to signatures. Observation 2: Traditional recursive descent parsers which need one function call per node are too expensive. 23 Efficient Parsing with State Machines Pre-construct parsing state machines based on parsing trees and vulnerability signatures. Studied eight protocols: HTTP, FTP, SMTP, eMule, BitTorrent, WINRPC, SNMP and DNS as well as their vulnerability signatures. Common relationship among leaf nodes. 24 Example for WINRPC Rectangles are states Parsing variables: R 0.. R instruction/byte for BIND PDU 25 Outline Motivation Feasibility Study: a Measurement Approach High Speed Parsing High Speed Matching for Large Rulesets. Evaluation Conclusions 26 A Matching Problem Example Data representations For all the vulnerability signatures we studied, we only need integers and strings Integer operators: ==, >, < String operators: ==, match_re(.,.), len(.). Example signature for Blaster worm 27 Matching Problem Formulation Suppose we have n signatures, each is defined on k matching dimensions (matchers) A matcher is a two-tuple (field, operation) or a four-tuple for the associate array elements. Efficiently report all the matched rules. Challenges for Single PDU matching problem (SPM) Large number of signatures n Large number of matchers k Large number of dont cares Cannot reorder matchers arbitrarily -- buffering constraint Field dependency Arrays, associate arrays Mutually exclusive fields. 28 Matching Algorithms Two steps 1.Pre-computation decides the rule order and matcher order 2.Divide-and-conquer comparison w/ matchers and combine the results efficiently Under each matcher m, parallel matching of all the rules that involve m Iteratively filter/combine the candidates from each matching. 29 Step 1: Pre-Computation Put the selective matchers earlier Observe buffering constraint & field arrival order 30 Step 2: Iterative Matching 31 Refinement and Extension SPM improvement Allow negative conditions Handle array case Handle associate array case Handle mutual exclusive case Report the matched rules as early as possible Extend to Multiple PDU Matching (MPM) Allow checkpoints. 32 Outline Motivation Feasibility Study: a Measurement Approach Problem Statement High Speed Parsing High Speed Matching for Large Rulesets. Evaluation Conclusions Evaluation Methodology Fully implemented and deployed to sniff a campus router hosting university Web servers and several labs. Run on a P4 3.8Ghz single core PC w/ 4GB memory. Much smaller memory usage. E.g., http 791 vulnerability sigs from 941 Snort rules: DFA: 5.29 GB vs. NetShield 1.08MB 33 34 Stress Test Results Traces from Tsinghua Univ. (TH) and Northwestern (NU) After TCP reassembly and preload the PDU in memory For DNS we only evaluate parsing. For WINRPC we have 45 vulnerability signatures which covers 3,519 Snort rules For HTTP we have 799 vulnerability signatures which covers 973 Snort rules. 35 Conclusions A novel network-based vulnerability signature matching engine Through measurement study on Snort ruleset, prove the vulnerability signature can improve most of the signatures in NIDS/IPS. Proposed parsing state machine for fast parsing Propose a candidate selection algorithm for matching a large number of vulnerability signature simultaneously 36 With Our Solutions Regular Expression Vulnerability AccuracyRelative Poor Much Better SpeedGoodEven faster MemoryOKBetter CoverageGoodSimilar Build a better Snort alternative 37 Backup Architecture of Network IDS Packet capture libpcap TCP reassembly Protocol identification Packet stream Signature matching (& protocol parsing when needed) 39 Observations Observation 1: Most matchers are good. After matching against them, only a small number of signatures can pass (candidates). String matchers are all good, and most integer matchers are good. We can buffer bad matchers to change the matching order. Observation 2: NIDS/NIPS will report all the matched rules regardless the ordering. Different from firewall rules. 40 Observation array PDU PDU parse tree Leaf nodes are integers or strings Vulnerability signature mostly based on leaf nodes Traditional recursive descent parsers (BINPAC) which need one function call per node are too expensive. Only need to parse the fields related to signatures 41 Limitations of Regular Expression Signatures Our network Traffic Filtering Internet Signature: 10.*01 X X Polymorphic attack (worm/botnet) might not have exact regular expression based signature Polymorphism! 42 Reason Regular expression is not power enough to capture the exact vulnerability condition! Cannot express exact condition Can express exact condition RE Shield X 43 Outline Motivation Feasibility Study: a measurement approach Problem Statement High Speed Parsing High Speed Matching for massive vulnerability Signatures. Evaluation Conclusions 44 What Do We Do? Build a NIDS/NIPS with much better accuracy and similar speed comparing with Regular Expression based approaches Feasibility: in Snort ruleset (6,735 signatures) 86.7% can be improved by vulnerability signatures. High speed Parsing: 2.7~12 Gbps High speed Matching: Efficient Algorithm for matching a large number of vulnerability rules HTTP, 791 vulnerability signatures at ~1Gbps 45 Network based IDS/IPS Accuracy (especially for IPS) False positive False negative Speed Coverage: Large ruleset Regular Expression Vulnerability AccuracyPoorMuch Better SpeedGood CoverageGood Regular expression is not power enough to capture the exact vulnerability condition!