Suricata and XDP, Performance with a S like Security · Kernel ask userspace for decision on...
Transcript of Suricata and XDP, Performance with a S like Security · Kernel ask userspace for decision on...
Suricata and XDP, Performance with a S like Security
É. Leblond
OISF
Nov. 29, 2018
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 1 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 1 / 43
Who am I?
Éric LeblondStamus Networks co-founder
Editor of a threat hunting solutionincluding Suricata based appliances
Netfilter core team memberReally low personal activity nowadays
[email protected] time member of OISFSuricata developer
In charge of packet acquisition1118 commits since 2010 (I like small patches)
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 2 / 43
About the journey
Adding bypass feature to Suricata2 years of development to see less and get more doneUsing kick ass technologies before their documentation has been written.
Figure: Summary of talk objectives
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 3 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 3 / 43
What it is not ?
https://twitter.com/randomuserid/status/1012474246503845888É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 4 / 43
A signature based IDS
From individual datagram to detectionGet packet per packetReconstruct to application layerRun detection engine
Key pointsGPLv2Owned by OISF foundation10 years oldScalability via multithreadingWritten in C and Rust
Example signature
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 5 / 43
Suricata NSM features
Supported protocolsProtocol analysis: http, ftp, smtp, tls, ssh smb, dcerpc, dns, nfs, ntp, ftp-data, tftp,ikev2, krb5, dhcpProtocol recognition: imap, msn
Log example
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 6 / 43
What it is ? or how to please developers
https://twitter.com/randomuserid/status/1012705279098490880
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 7 / 43
File related features
File analysisMagic computation and in file data matchChecksum computation and file extraction to diskSupported protocols: http, smtp, smb, ftp, nfs
Fileinfo example
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 8 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 8 / 43
Suricata live modes
Intrusion Detection SystemAF_PACKET capture method under LinuxGet raw packet from cardDo complete analysis
Intrusion Prevention System
Netfilter with NFQUEUE on LinuxKernel ask userspace for decision onpackets
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 9 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 9 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 9 / 43
Suricata reconstruction and normalization
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 10 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 10 / 43
Impact of loosing packets
MethodologyUse a sample trafficModify the pcap file to have specified random packet lossDo it 3 times par packet lossGet graph out of that
Test dataUsing a test pcap of 445Mo.Real traffic but lot of malicious behaviorsTraffic is a bit old
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 11 / 43
Alert loss by packet loss
Some numbers10% missed alerts with 3% packets loss50% missed alerts with 25% packets loss
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 12 / 43
The case of file extraction
Some numbers10% failed file extraction with 0.4% packets loss50% failed file extraction with 5.5% packets loss
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 13 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 13 / 43
The elephant flow problem (1/2)
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 14 / 43
The elephant flow problem (2/2)
Ring buffer overrunLimited sized ring bufferOverrun cause packets lossthat cause streaming malfunction
Ring size increaseWork aroundUse memoryFail for non burst
Dequeue at NQueue at speed N+M
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 15 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 15 / 43
Stream depth method
Attacks characteristicIn most cases attack is done at start of TCP sessionGeneration of requests prior to attack is not commonMultiple requests are often not even possible on same TCP session
Stream reassembly depthReassembly is done till stream.reassembly.depth bytes.Stream is not analyzed once limit is reachedIndividual packet continue to be inspected
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 16 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 16 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 16 / 43
Introducing bypass
Stop packet handling as soon as possibleTag flow as bypassedMaintain table of bypassed flowsDiscard packet if part of a bypassed flow
Bypass methodLocal bypass: Suricata discard packet after decodingCapture bypass: capture method maintain flow table and discard packets ofbypassed flows
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 17 / 43
Bypassing big flow: local bypass
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 18 / 43
Bypassing big flow: capture bypass
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 19 / 43
Implementation
Suricata updateAdd callback functionCapture method register itself and provide a callbackSuricata calls callback when it wants to offload
NFQ bypass in Suricata 3.2Update capture register functionWritten callback function
Set a mark with respect to a mask on packetMark is set on packet when issuing the verdict
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 20 / 43
Suricata NFQ and bypass
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 21 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 21 / 43
Stream depth bypass
Stop all treatment after bypassGo beyond what is currently doneDisable individual packet treatment once stream depth is reached
Activating stream depth bypassSet stream.bypass to yes in YAML
TLS bypassencrypt-handling: bypass
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 22 / 43
Selective bypass
Ignore some trafficIgnore intensive traffic like NetflixCan be done independently of stream depthCan be done using generic or custom signatures
The bypass keywordA new bypass signature keywordTrigger bypass when signature matchExample of signature
pass h t t p any any −> any any ( content : " s u r i c a t a . i o " ; \ \h t tp_hos t ; bypass ; s id :6666; rev : 1 ; )
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 23 / 43
And now AF_PACKET bypass
What’s neededSuricata to tell kernel to ignore flowsKernel system able to
Maintain a list of flow entriesDiscard packets belonging to flows in the listUpdate from userspace
eBPF filter using mapseBPF introduce mapsDifferent data structures
Hash, array, . . .Update and fetch from userspace
Looks good!
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 24 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 24 / 43
Extended Berkeley Packet Filter
Berkeley Packet FilterVirtual machine inside kernelArithmetic operations and tests on the packet dataFilters are injected by userspace in kernel via syscall
Extended BPFExtended virtual machine: more operators, data and function accessVarious attachment points
SocketSyscallTraffic control
Kernel and userspace shared structuresHash tablesArrays
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 25 / 43
LLVM backend
From C file to eBPF codeWrite C codeUse eBPF LLVM backend (since LLVM 3.7)Use libbpf
Get ELF fileExtract and load section in kernel
BCC: BPF Compiler collectionInject eBPF into kernel from high level scripting languageTrace syscalls and kernel functionshttps://github.com/iovisor/bcc
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 26 / 43
BCC tracing tools
http://www.brendangregg.com/ebpf.htmlÉ. Leblond (OISF) Suricata and XDP Nov. 29, 2018 27 / 43
eBPF applied to security
AdvantagesReally extensibleKernel version independant when not intercepting functionsExtract info all system stacks
Host security monitoring at NetflixLinux Monitoring at Scale with eBPF (Brendan Gregg & Alex Maestretti)https://youtu.be/44nV6Mj11uw
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 28 / 43
Example: BCC socket bind 1/2
Detect network serversGet all bind call to detect servicesOutput result to console
A BCC scriptPython codeeBPF code as C in a string
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 29 / 43
Example: BCC socket bind 2/2
DemoStart sobindStart a nc command to listen to port 2233
Outputsudo python ./sobindPID COMM PROTO PORT ADDR9565 nc TCPv4 2233 0.0.0.09572 nc TCPv4 2233 127.0.0.2
Key featuresDirect hook in the system callNo /proc scanning but polling of resultsGet it there:https://gist.github.com/regit/1e591311fa3ba5cd0b8d73940348599a
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 30 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 30 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 30 / 43
Suricata eBPF bypass architecture
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 31 / 43
Test methodology
Test setupIntel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHzIntel Corporation 82599ES 10-Gigabit SFI/SFP+Live traffic:
Around 1Gbps to 2GbpsReal users so not reproducible
TestsOne hour long runDifferent stream depth valuesCollected Suricata statistics counters (JSON export)Graphs done via Timelion(https://www.elastic.co/blog/timelion-timeline)
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 32 / 43
Results: stream bypass at 512kb
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 33 / 43
A few words on graphics
Tests at 512kbWe have on big flow that kill thebandwidthCapture get almost nullEven number of closed bypassedflows is low
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 34 / 43
Results
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 35 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 35 / 43
A Linux kernel feature
Run a eBPF code the earliest possiblein the driverin the cardbefore the regular kernel path
Act on dataDrop packet (eXtreme Drop Performance)Transmit to kernelRewrite and transmit packet to kernelRedirect to another interfaceCPU load balance
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 36 / 43
Implementation
Similar to eBPF filterSame logic for bypassOnly verdict logic is different
But annoying differenceeBPF code does the parsingNeed to bind to an interface
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 37 / 43
Suricata XDP architecture
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 38 / 43
AF_PACKET IPS mode
IPS and bypassCan’t drop packet at bypass stageNeed to forward from one iface to another
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 39 / 43
XDP and IPS mode: bypass
Use TX_REDIRECTDirect copy from interface to interface
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 40 / 43
Results
Direct NIC to NIC transferSkip all kernel taskWire speed copyIf eBPF code is fast enough
Obtained performanceTODO: Ask OISF marketing for some fake numbers to show
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 41 / 43
1 IntroductionSuricata 101Suricata on live traffic
2 ProblemReconstruction workPacket loss impactElephant flowWork less to get more
3 Suricata BypassIntroducing bypassBypass strategy
4 Extended Berkeley Packet Filter5 AF_PACKET bypass
eBPF bypassXDP support
6 Conclusion
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 41 / 43
Conclusion
Suricata, eBPF and XDPAvailable in Suricata 4.1, need Linux 4.16Network card bypass for Netronome comingAF_XDP capture is now in Linux vanilla
More informationStamus Networks: https://www.stamus-networks.com/Septun II: https://github.com/pevma/SEPTun-Mark-II/Suricata doc: http://suricata.readthedocs.io/en/latest/capture-hardware/ebpf-xdp.html
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 42 / 43
Questions ?
Thanks toJesper Dangaard BrouerAlexei StarovoitovDaniel Borkmann
Contact [email protected]: @regiteric
Want more fun ?Come to Suricata trainings: https://suricata-ids.org/training/
Suricon: https://suricon.net/
É. Leblond (OISF) Suricata and XDP Nov. 29, 2018 43 / 43