Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap...
Transcript of Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap...
![Page 1: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/1.jpg)
Robin Sommer International Computer Science Institute, &
Corelight, Inc.
[email protected] [email protected]
http://www.icir.org/robin
Spicy: A Unified Deep Packet Inspection Framework Dissecting All Your Data
![Page 2: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/2.jpg)
Deep Packet Inspection
2
Tap
IDS
InternetLocal
Network
![Page 3: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/3.jpg)
Deep Packet Inspection
2
Tap
IDS
1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash and compare with database. 4. Report, and potentially kill, if found.
Example: Finding downloads of known malware.
InternetLocal
Network
![Page 4: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/4.jpg)
Deep Packet Inspection
2
Tap
IDS
1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash and compare with database. 4. Report, and potentially kill, if found.
Example: Finding downloads of known malware.
InternetLocal
Network
![Page 5: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/5.jpg)
Protocol Parsing
3
Request for /x/y/foo.zip
Status OK plus data 5.6.7.8/801.2.3.4/4321
Web Server
Web Client
![Page 6: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/6.jpg)
Protocol Parsing
3
Request for /x/y/foo.zip
Status OK plus data 5.6.7.8/801.2.3.4/4321
Web Server
Web Client
ACK FIN FIN
TCP connection established
...SYN SYN ACK ACK ACK ...
![Page 7: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/7.jpg)
Protocol Parsing
3
Request for /x/y/foo.zip
Status OK plus data 5.6.7.8/801.2.3.4/4321
TCP stream reassembly for originator
Request for /x/y/foo.zip, protocol version 1.1, HTTP headers
GET /x/y/foo.zip HTTP/1.1 …
Web Server
Web Client
ACK FIN FIN
TCP connection established
...SYN SYN ACK ACK ACK ...
![Page 8: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/8.jpg)
Protocol Parsing
3
Request for /x/y/foo.zip
Status OK plus data 5.6.7.8/801.2.3.4/4321
TCP stream reassembly for originator
Request for /x/y/foo.zip, protocol version 1.1, HTTP headers
GET /x/y/foo.zip HTTP/1.1 …
TCP stream reassembly for responder
Reply with page content for further analysis (e.g., hash; unpack & parse files)
200 OK …
Web Server
Web Client
ACK FIN FIN
TCP connection established
...SYN SYN ACK ACK ACK ...
![Page 9: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/9.jpg)
Protocol Parsing
3
Request for /x/y/foo.zip
Status OK plus data 5.6.7.8/801.2.3.4/4321
TCP connection tear down
TCP stream reassembly for originator
Request for /x/y/foo.zip, protocol version 1.1, HTTP headers
GET /x/y/foo.zip HTTP/1.1 …
TCP stream reassembly for responder
Reply with page content for further analysis (e.g., hash; unpack & parse files)
200 OK …
Web Server
Web Client
ACK FIN FIN
TCP connection established
...SYN SYN ACK ACK ACK ...
![Page 10: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/10.jpg)
Parsing Is Hard
4
...SYN SYN ACK ACK ACK ACK FIN FIN...
![Page 11: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/11.jpg)
Parsing Is Hard
Must be robust Lots of “crud” in real-world networks Cannot trust input
4
...SYN SYN ACK ACK ACK ACK FIN FIN...
![Page 12: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/12.jpg)
Parsing Is Hard
Must be robust Lots of “crud” in real-world networks Cannot trust input
Must be efficient 100,000s of concurrent connections Incremental processing for low latency & memory usage
4
...SYN SYN ACK ACK ACK ACK FIN FIN...
![Page 13: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/13.jpg)
Parsing Is Hard
Must be robust Lots of “crud” in real-world networks Cannot trust input
Must be efficient 100,000s of concurrent connections Incremental processing for low latency & memory usage
Must be complete Leaving out parts of the protocol opens evasion opportunities Protocols can be really complex (SMB …)
4
...SYN SYN ACK ACK ACK ACK FIN FIN...
![Page 14: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/14.jpg)
There are a lot of protocols out there …
5
Even a simple case involves 5 protocols
PCAP
IP
TCP
HTTP
Ethernet
![Page 15: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/15.jpg)
There are a lot of protocols out there …
5
Even a simple case involves 5 protocols
PCAP
IP
TCP
HTTP
A few popular protocols account for the bulk of traffic in most environments
(e.g., TCP/IP, HTTP, TLS, DNS, SMTP, IMAP)Ethernet
![Page 16: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/16.jpg)
There are a lot of protocols out there …
5
Even a simple case involves 5 protocols
PCAP
IP
TCP
HTTP
A few popular protocols account for the bulk of traffic in most environments
(e.g., TCP/IP, HTTP, TLS, DNS, SMTP, IMAP)
Long tail of further protocols, often environment-specific
(e.g., SMB, Modbus, BACnet, more L2)
Ethernet
![Page 17: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/17.jpg)
There are a lot of protocols out there …
5
Even a simple case involves 5 protocols
PCAP
IP
TCP
HTTP
A few popular protocols account for the bulk of traffic in most environments
(e.g., TCP/IP, HTTP, TLS, DNS, SMTP, IMAP)
Long tail of further protocols, often environment-specific
(e.g., SMB, Modbus, BACnet, more L2)
Ethernet
File formats amplify the challenge
![Page 18: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/18.jpg)
Example: Bro 2.5
6
AYIYABitTorrentDCE_RPCDHCPDNP3DNSDTLSFTP
FingerGTPv1
GnutellaHTTPICMP
IPv4/6IRC
Ident
KerberosLoginModbusMySQLNCPNFSNTP
NetBIOSPE
POP3Portmapper
RadiusRDP
RloginRshSMB
SIPSMTPSNMPSOCKSSSHSSL
SyslogTCP
TelnetTeredoUDPX509ZIP
![Page 19: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/19.jpg)
A Tale of Three Open-Source IDS
7
Suricata
![Page 20: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/20.jpg)
A Tale of Three Open-Source IDS
7
Suricata
Shared parsers?
Every DPI application rewrites its parsers — usually in C/C++!
None.
![Page 21: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/21.jpg)
Opportunity: Provide Platform for Parsers
8
![Page 22: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/22.jpg)
Opportunity: Provide Platform for Parsers
Protocols leverage a rather small set of patterns Readable line-based formats for text protocols Static “prototocol data units” (PDU) for binary protocols Request/response structure Common sub-formats (HTTP/MIME/ASN.1) Fragmentation (even at app layer!)
8
![Page 23: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/23.jpg)
Opportunity: Provide Platform for Parsers
Protocols leverage a rather small set of patterns Readable line-based formats for text protocols Static “prototocol data units” (PDU) for binary protocols Request/response structure Common sub-formats (HTTP/MIME/ASN.1) Fragmentation (even at app layer!)
But: Potpourri of protocols remains diverse still Every protocol does something different
8
![Page 24: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/24.jpg)
Opportunity: Provide Platform for Parsers
Protocols leverage a rather small set of patterns Readable line-based formats for text protocols Static “prototocol data units” (PDU) for binary protocols Request/response structure Common sub-formats (HTTP/MIME/ASN.1) Fragmentation (even at app layer!)
But: Potpourri of protocols remains diverse still Every protocol does something different
Can we leverage similarities, while remaining flexible?
8
![Page 25: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/25.jpg)
Opportunity: Provide Platform for Parsers
Protocols leverage a rather small set of patterns Readable line-based formats for text protocols Static “prototocol data units” (PDU) for binary protocols Request/response structure Common sub-formats (HTTP/MIME/ASN.1) Fragmentation (even at app layer!)
But: Potpourri of protocols remains diverse still Every protocol does something different
Can we leverage similarities, while remaining flexible?
Can we reuse code across applications?
8
![Page 26: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/26.jpg)
Meanwhile, in another domain …
9
There are powerful tools for implementing parsers for programming languages.
![Page 27: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/27.jpg)
Meanwhile, in another domain …
9
There are powerful tools for implementing parsers for programming languages.
exp: NUM { $$ = $1; } | exp '+' exp { $$ = $1 + $2; } | exp ‘-' exp { $$ = $1 - $2; } | exp ‘*' exp { $$ = $1 * $2; } | exp ‘/' exp { $$ = $1 / $2; }
![Page 28: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/28.jpg)
Meanwhile, in another domain …
9
There are powerful tools for implementing parsers for programming languages.
exp: NUM { $$ = $1; } | exp '+' exp { $$ = $1 + $2; } | exp ‘-' exp { $$ = $1 - $2; } | exp ‘*' exp { $$ = $1 * $2; } | exp ‘/' exp { $$ = $1 / $2; }
yyparse() Host ApplicationYacc
![Page 29: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/29.jpg)
Meanwhile, in another domain …
9
There are powerful tools for implementing parsers for programming languages.
exp: NUM { $$ = $1; } | exp '+' exp { $$ = $1 + $2; } | exp ‘-' exp { $$ = $1 - $2; } | exp ‘*' exp { $$ = $1 * $2; } | exp ‘/' exp { $$ = $1 / $2; }
yyparse() Host ApplicationYacc
These parsers aren’t suitable for DPI, unfortunately.
No support for concurrent, incremental processing No support for domain-specific idioms
![Page 30: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/30.jpg)
Domain-specific Parser Generation
10
IMC 2006
![Page 31: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/31.jpg)
Domain-specific Parser Generation
10
IMC 2006
type ClientHello(rec: HandshakeRecord) = record { client_version: uint16; gmt_unix_time : uint32; random_bytes : bytestring &length = 28; session_len : uint8; session_id : uint8[session_len]; dtls_cookie : case client_version of { DTLSv10, DTLSv12 -> cookie : ClientHelloCookie(rec); default -> nothing: bytestring &length=0; }; […] } TLS v3 Client Hello (Source: Bro’s TLS analyzer)
![Page 32: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/32.jpg)
Domain-specific Parser Generation
10
class binpac:: ConnectionAnalyzer
Host ApplicationBinPAC
IMC 2006
type ClientHello(rec: HandshakeRecord) = record { client_version: uint16; gmt_unix_time : uint32; random_bytes : bytestring &length = 28; session_len : uint8; session_id : uint8[session_len]; dtls_cookie : case client_version of { DTLSv10, DTLSv12 -> cookie : ClientHelloCookie(rec); default -> nothing: bytestring &length=0; }; […] } TLS v3 Client Hello (Source: Bro’s TLS analyzer)
![Page 33: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/33.jpg)
Domain-specific Parser Generation
10
class binpac:: ConnectionAnalyzer
Host ApplicationBinPAC
IMC 2006
type ClientHello(rec: HandshakeRecord) = record { client_version: uint16; gmt_unix_time : uint32; random_bytes : bytestring &length = 28; session_len : uint8; session_id : uint8[session_len]; dtls_cookie : case client_version of { DTLSv10, DTLSv12 -> cookie : ClientHelloCookie(rec); default -> nothing: bytestring &length=0; }; […] } TLS v3 Client Hello (Source: Bro’s TLS analyzer)
BinPAC works, but solves the problem only partially.
Remains limited to syntax, cannot express logic. Still needs custom C++ for logic & integration.
Remains limited to app protocols & connection structure. Lacks support for higher-level idioms.
![Page 34: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/34.jpg)
New Framework: Spicy
11
Integrates experience from many years of writing parsers manually and with BinPAC.
![Page 35: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/35.jpg)
New Framework: Spicy
Expresses both syntax and logic
Supports protocols and file formats
Facilitates composition and reuse
Supports error handling and recovery
Just-in-time compilation via LLVM
11
Integrates experience from many years of writing parsers manually and with BinPAC.
![Page 36: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/36.jpg)
Spicy Example: Parsing SMTP Banners
12
220 mx.foo.com ESMTP Postfix
![Page 37: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/37.jpg)
Spicy Example: Parsing SMTP Banners
12
220 mx.foo.com ESMTP Postfix
module SMTP;
export type Greeting = unit { : /220 +/; domain : /[^ ]+/; : / */; protocol: /(E?SMTP)?/; : / */; software: /[^ ]*/; on %done { print self; }} smtp.spicy
![Page 38: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/38.jpg)
Spicy Example: Parsing SMTP Banners
12
220 mx.foo.com ESMTP Postfix
# echo "220 mx.foo.com ESMTP Postfix” | spicy-driver smtp.spicy<domain=mx.foo.com, protocol=ESMTP, software=Postfix>
module SMTP;
export type Greeting = unit { : /220 +/; domain : /[^ ]+/; : / */; protocol: /(E?SMTP)?/; : / */; software: /[^ ]*/; on %done { print self; }} smtp.spicy
![Page 39: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/39.jpg)
Host Application API
13
# Compile Spicy code just-in-time (C++)
auto ctx = new spicy::CompilerContext();
atuo llvm_module = ctx->compile(“smtp.spicy”);
auto linked_module = ctx->linkModules("SMTP", llvm_module);
auto jit = ctx->jit(linked_module);
auto parse_func = jit->nativeFunction(“smtp_greeting_parse”)auto resume_func = jit->nativeFunction(“smtp_greeting_resume”)
![Page 40: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/40.jpg)
Host Application API
13
# Compile Spicy code just-in-time (C++)
auto ctx = new spicy::CompilerContext();
atuo llvm_module = ctx->compile(“smtp.spicy”);
auto linked_module = ctx->linkModules("SMTP", llvm_module);
auto jit = ctx->jit(linked_module);
auto parse_func = jit->nativeFunction(“smtp_greeting_parse”)auto resume_func = jit->nativeFunction(“smtp_greeting_resume”)
# Feed data into parser (C).
hlt_bytes* data = hlt_bytes_new_from_data(“220 mx.foo.”);void* cookie = (*parse_func)(data);
hlt_bytes* next = hlt_bytes_new_from_data(“.com ESMTP Postfix”);hlt_bytes_append(data, next);cookie = (*resume_func)(cookie);
![Page 41: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/41.jpg)
A File Format: Tar
14
![Page 42: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/42.jpg)
A File Format: Tar
14
module tar;
export type Archive = unit { files: list<File>; : uint<8>(0x0); : bytes &length=511;};
type File = unit { header: Header; data : bytes &length=self.header.size; : bytes &length=512-(self.header.size mod 512)};
type Type = enum { REG=0, LNK=1, SYM=2, CHR=3, BLK=4, DIR=5, FIFO=6};
type Header = unit { name : bytes &length=100; mode : bytes &length=8; uid : bytes &length=8; gid : bytes &length=8; size : bytes &length=12 &convert=$$.to_uint(8); mtime : bytes &length=12 &convert=$$.to_time(8); chksum: bytes &length=8 &convert=$$.to_uint(8); tflag : bytes &length=1 &convert=$$.to_uint(8); lname : bytes &length=100; : bytes &length=88; # Skip further fields prefix: bytes &length=155; : bytes &length=12;.
var full_path: bytes;
on %done { if ( ! self.tflag ) self.tflag = Type::REG; self.full_path = self.prefix + b"/"+ self.name; }
![Page 43: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/43.jpg)
A File Format: Tar
14
module tar;
export type Archive = unit { files: list<File>; : uint<8>(0x0); : bytes &length=511;};
type File = unit { header: Header; data : bytes &length=self.header.size; : bytes &length=512-(self.header.size mod 512)};
type Type = enum { REG=0, LNK=1, SYM=2, CHR=3, BLK=4, DIR=5, FIFO=6};
type Header = unit { name : bytes &length=100; mode : bytes &length=8; uid : bytes &length=8; gid : bytes &length=8; size : bytes &length=12 &convert=$$.to_uint(8); mtime : bytes &length=12 &convert=$$.to_time(8); chksum: bytes &length=8 &convert=$$.to_uint(8); tflag : bytes &length=1 &convert=$$.to_uint(8); lname : bytes &length=100; : bytes &length=88; # Skip further fields prefix: bytes &length=155; : bytes &length=12;.
var full_path: bytes;
on %done { if ( ! self.tflag ) self.tflag = Type::REG; self.full_path = self.prefix + b"/"+ self.name; }
# tar tvf mp.tarfoobar/staff 0 2016-05-15 18:58 mp/foobar/staff 39548 2016-05-15 18:58 mp/part01.txtfoobar/staff 39503 2016-05-15 18:58 mp/part02.txt*/
# cat print-tar.spicymodule PrintTar;
import tar;
on tar::Archive::%done { print self.files;}
# cat mp.tar | spicy-driver tar.spicy print-tar.spicy[<header=<name=b"mp/", mode=b"000755", uid=b"000771", gid=b"000024", size=0, mtime=2016-05-16T02:58:19Z, chksum=5100, tflag=DIR>, data=b””, […], full_path=b”mp/“>]
[<header=<name=b"mp/part01.txt", mode=b"000644", uid=b"000771", gid=b"000024", size=39548, mtime=2016-05-16T02:58:19Z, chksum=6351, tflag=REG>, data=b"A seashore. Some way out to sea […]”, […], full_path=b”mp/part01.txt“>]
[<header=<name=b”mp/part02.txt", mode=b"000644", uid=b"000771", gid=b"000024", size=39503, mtime=2016-05-16T02:58:11Z, chksum=6348, tflag=REG>, data=b"A man appears on the top of a sand […]”, […] full_path=b”mp/part02.txt“>]
![Page 44: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/44.jpg)
Composition: Pipelining Layers
15
![Page 45: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/45.jpg)
Composition: Pipelining Layers
15
type HTTP::Body = unit(msg: Message, delivery_mode: DeliveryMode) {
var data: sink;
on %init { # Add parser for body content (e.g., application/x-tar) self.data.connect_mime_type(msg.content_type);
if ( msg.content_encoding == b"gzip" ) { self.data.add_filter(Spicy::Filter::GZIP); }
switch ( delivery_mode ) { DeliveryMode::EndOfData -> : bytes &eod -> self.data; DeliveryMode::Length -> : bytes &length=msg.content_length -> self.data; DeliveryMode::Multipart -> : list<[^\r\n]*\r?\n/> &until($$ == msg.boundary) foreach { self.data.write($$); }};
![Page 46: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/46.jpg)
Error Recovery
16
![Page 47: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/47.jpg)
Error Recovery
16
type HTTP::Requests = unit { requests: list<Request> &synchronize;};
type HTTP::Request = unit { request: RequestLine; message: Message;};
type HTTP::RequestLine = unit { %synchronize-at = /^(GET|POST|HEAD) /; method: Token; : WhiteSpace; uri: Token; : WhiteSpace; : /HTTP\//; version: /[0-9]+\.[0-9]*/; : NewLine;};
type HTTP::Message = unit { ... }
![Page 48: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/48.jpg)
Evaluation: Writing Spicy Parsers
17
![Page 49: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/49.jpg)
Evaluation: Writing Spicy Parsers
17
![Page 50: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/50.jpg)
Evaluation: Writing Spicy Parsers
17
PCAP
Ethernet
HTTP
PCAP
MS-Cert
BACnet
IPv4
UDP
TCP
DNS
TLS X.509
![Page 51: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/51.jpg)
Evaluation: Writing Spicy Parsers
17
PCAP
Ethernet
HTTP
PCAP
MS-Cert
BACnet
IPv4
UDP
TCP
DNS
TLS X.509
Trace 1X.509
MS Cert StoreHTTPTCPIP
EthernetPCAP
Trace 2X.509TLSTCPIP
EthernetPCAPHTTPTCPIP
EthernetPCAP
![Page 52: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/52.jpg)
Evaluation: Real-world Performance
18
DNS: Full Berkeley port 53 traffic. 1GB trace, 10min, 65M messages.
HTTP: 1/25 of Berkeley port 80 traffic. 30GB trace, 52min, 340k messages.
Add Spicy plugin for Bro to compare parsing with a native Bro.
Traces:
![Page 53: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/53.jpg)
Evaluation: Real-world Performance
18
DNS: Full Berkeley port 53 traffic. 1GB trace, 10min, 65M messages.
HTTP: 1/25 of Berkeley port 80 traffic. 30GB trace, 52min, 340k messages.
Correctness
Spicy captures protocols correctly.
Add Spicy plugin for Bro to compare parsing with a native Bro.
Traces:
![Page 54: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/54.jpg)
Evaluation: Real-world Performance
18
DNS: Full Berkeley port 53 traffic. 1GB trace, 10min, 65M messages.
HTTP: 1/25 of Berkeley port 80 traffic. 30GB trace, 52min, 340k messages.
Correctness
Spicy captures protocols correctly.
Add Spicy plugin for Bro to compare parsing with a native Bro.
Let’s see.
Performance
Traces:
![Page 55: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/55.jpg)
Performance: Spicy vs. C++ in Bro
19
0.0B 0.2B 0.4B 0.6B 0.8B 1.0B 1.2B 1.4B 1.6B 1.8BC38 cycOHs
6tanGaUG
HIL7I
6tanGaUG
HIL7I
1567G683G 643G 241G
1580G852G 450G 21G 258G
712G177G 356G 180G
1173G469G 405G 81G 217G
HTTP
DNSProtocol PDrsing
6cULSt ExHcutLon
HIL7I-to-BUo GOuH
2thHU
SpicySpicy
Spicy
Spicy
![Page 56: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/56.jpg)
Performance: Spicy vs. C++ in Bro
19
0.0B 0.2B 0.4B 0.6B 0.8B 1.0B 1.2B 1.4B 1.6B 1.8BC38 cycOHs
6tanGaUG
HIL7I
6tanGaUG
HIL7I
1567G683G 643G 241G
1580G852G 450G 21G 258G
712G177G 356G 180G
1173G469G 405G 81G 217G
HTTP
DNSProtocol PDrsing
6cULSt ExHcutLon
HIL7I-to-BUo GOuH
2thHU
SpicySpicy1.25x
Spicy2.65x
Spicy
![Page 57: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/57.jpg)
Bro Integration: “3rd Generation Parsers”
20
![Page 58: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/58.jpg)
Bro Integration: “3rd Generation Parsers”
Generation 1: Manually written C++ code.
20
![Page 59: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/59.jpg)
Bro Integration: “3rd Generation Parsers”
Generation 1: Manually written C++ code.
Generation 2: BinPAC - “yacc for protocols”.
20
![Page 60: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/60.jpg)
Bro Integration: “3rd Generation Parsers”
Generation 1: Manually written C++ code.
Generation 2: BinPAC - “yacc for protocols”.
Generation 3: Spicy - A “closed” system.
20
Spicy Grammar*.spicy
Event Definitions*.evt
BroBro
Scripts*.bro
JIT
![Page 61: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/61.jpg)
Advanced Spicy Features
Composibility
Error detection & recovery
Protocol detection
Reassembly/defragmentation
Generating wire format
21
![Page 62: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/62.jpg)
Implementation: HILTI Toolchain
22
IMC 2014
![Page 63: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/63.jpg)
Implementation: HILTI Toolchain
22
Spicy
Runtime Library
Spicy Compiler
Spicy Grammar
LLVM Bitcode
Compiler/Linker
HILTI Compiler
HILTI Environment LLVM Toolchain
Runtime Library
C Interface
Stubs
Machine Code
HILTI Machine
Code
IMC 2014
![Page 64: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/64.jpg)
The HILTI Model
23
Performance via Abstraction
Transparent improvement under the hood Integration of non-standard hardware High-level, global compiler optimizations Automatic parallelization
Means and glue to share functionality HILTI library of common high-level components
Facilitating Reuse
Secure Execution
EnvironmentSandboxed execution Automatic memory management
![Page 65: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/65.jpg)
Summary
24
Spicy is a next-generation parser generator for deep packet inspection systems.
![Page 66: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/66.jpg)
Summary
24
Expresses both syntax and semantics
Supports protocols and file formats
Facilitates composition and reuse
Supports error handling and recovery
Just-in-time compilation via LLVM
Spicy is a next-generation parser generator for deep packet inspection systems.
![Page 67: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/67.jpg)
Summary
24
Expresses both syntax and semantics
Supports protocols and file formats
Facilitates composition and reuse
Supports error handling and recovery
Just-in-time compilation via LLVM
Spicy is a next-generation parser generator for deep packet inspection systems.
http://www.icir.org/hilti
Open-source, BSD-licensed prototype.
![Page 68: Spicy: A Unified Deep Packet Inspection Framework ... · PDF fileDeep Packet Inspection 2 Tap IDS 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash](https://reader031.fdocuments.in/reader031/viewer/2022030408/5a8a25217f8b9a7f398b9551/html5/thumbnails/68.jpg)
25
The Bro Project www.bro.org [email protected]
@Bro_IDS
Professional Bro Solutions www.corelight.io [email protected] @corelight_inc
Corelight is hiring!
Robin SommerInternational Computer Science Institute, &
Corelight, Inc.
[email protected] [email protected]
http://www.icir.org/robin
Questions?