Sniffing network traffic in Python

Jose Nazario, Ph.D. <jose@monkey.org>

Why Python?

• Interpreted language– Bound to be slower than C

• Rapid development• Easy data structure use• Fewer LoC per tool• Easy to manipulate strings• http://www.python.org/

Marrying Python and Sniffing

• Librares in C– Often SWIGged, exported to Python– pcap, dnet, nids …

• Modules – pypcap/pcappy – pcap for python– dpkt – packet deconstruction library– libdnet – packet construction library (has python

bindings in the distribution)

– pynids – connection reassembly tool

libnids – reassemble IP streams

NIDS “E” box (event generation box)Userland TCP/IP stackBased on Linux 2.0.36 IP stackUses libpcap, libnet internallyIP fragment reassembly

Kernel

IP stack

Userland

Kernel

IP stack

Userland

IP stackLibnids

libnids Basics• Initialize

– nids_init()• Register callbacks

– nids_register_tcp()– nids_regster_ip()– nids_regiser_udp()

• Run!– nids_run()

• React– nids_kill_tcp()

nids_run()

UDP callbackTCP callback IP callback

TCP stream object: - TCP state - client data - server data - source IP, port - dest IP, port - seq, ack, etc …

UDP packet: - source IP, port - dest IP, port - UDP payload

IP packet - struct IP packet - contains upper layers

libnids TCP states• NIDS_JUST_ESTABLISHED

– New TCP connected state (3WHS)– Must set stream->{client,server}.collect=1

to get stream payload collected• NIDS_DATA

– Data within a known, established TCP connection• NIDS_RESET, NIDS_CLOSE, NIDS_TIMED_OUT– TCP connection is reset, closed gracefully, or was lost

libnids doesn’t expose SYN_SENT, FIN_WAIT, etc …

pynids Basics

• Event driven interface (nids_run(), nids_next())– TCP stream reassembly– TCP state exposure– Creates a TCP object

• Holds addresses, data, etc – UDP and IP packet reassembly

Basic pynids Steps• Initialize

– nids_init()• Establish parameters

– nids.param(“attribute”, value)• Register callbacks

– nids.register_tcp(handleTcp)– def handleTcp(tcp): …

• Go!– nids_run()– while 1: nids_next()

pynids Order of Operations

• Packets come in

• TCP?– State exist? Create state or reuse state– Append data– Process based on state in callback

• UDP or IP?– Use handler, pass packet in– You process in callback

Code Example (Python)import nids<handleTcpStream>

def main(): nids.param("scan_num_hosts", 0) if not nids.init(): print "error -", nids.errbuf() sys.exit(1) nids.register_tcp(handleTcpStream) try: nids.run() # loop forever except KeyboardInterrupt: sys.exit(1)

Code Example (Python) contdef handleTcpStream(tcp): if tcp.nids_state == nids.NIDS_JUST_EST: if dport in (80, 8000, 8080): tcp.client.collect = 1 tcp.server.collect = 1 elif tcp.nids_state == nids.NIDS_DATA: tcp.discard(0) elif tcp.nids_state in end_states: print "addr:", tcp.addr # may be binary print "To server:“, tcp.server.data print "To client:“, tcp.client.data

Code Example (C)int main(int argv, char *argv[]) { if (nids_init() == 0) err(1, “error, %s”, nids_errbuf); nids_register_tcp(handleTcp); nids_run(); exit(0);}

Code Example (C), contint handleTcp(struct tcp_stream *tcp) { switch (tcp->nids_state) { case ‘NIDS_JUST_EST’: if ((tcp->addr.dest == 80) || (tcp->addr.dest == 8000) || (tcp->addr.dest == 8080) { tcp.server.collect = 1; tcp.client.collect = 1; } break; case ‘NIDS_DATA’: nids_discard(tcp, 0); break; case ‘NIDS_CLOSE’: case ‘NIDS_RESET’: case ‘NIDS_TIMED_OUT’: printf(“((%s, %d), (%s, %d))\n”, inet_ntoa(tcp->saddr), tcp.srce, inet_ntoa(tcp->daddr), tcp.dest); printf(“%s\n”, tcp->server.data); printf(“%s\n”, tcp->client.data); break; }}

About the same LoC, until we start string manipulation

VersionDetect

• Small python tool• Reports on headers• Fully passive

– Support for: SSH (client, server), WWW (client, server), and SMTP clients

• Motivation: coordinate data collection with TCP stack fingerprinting

63.236.16.161 SymbianOS 6048 (on Nokia 7650?) www 80/tcp 63.236.16.161: 80: Microsoft-IIS/6.0

VersionDetect Output 192.168.1.7: 22: SSH-2.0-OpenSSH_3.5 192.168.1.101:http: Mozilla/5.0 (X11; U; OpenBSD i386; en-

US; rv:1.5a) Gecko/20031030 Mozilla Firebird/0.6.1 168.75.65.85: 80: Microsoft-IIS/5.0 165.1.76.60: 80: Netscape-Enterprise/3.6 SP2 168.75.65.69: 80: Microsoft-IIS/5.0 168.75.65.87: 80: Microsoft-IIS/5.0 69.28.159.7: 80: ZEDO 3G 198.65.148.234: 80: Apache/1.3.29 (Unix) PHP/4.3.3 216.150.209.231: 80: Apache/1.3.31 (Unix) 212.187.153.30: 80: Apache/1.3.31 (Unix) 212.187.153.37: 80: Apache/1.3.31 (Unix) 212.187.153.32: 80: thttpd/2.25b 29dec2003 64.209.232.207: 80: Apache/1.3.27 (Unix) mod_perl/1.27 216.239.39.99: 80: CAFE/1.0

http-graph

• Small, passive python tool• Examines HTTP request header:

GET /blog/styles-site.css HTTP/1.1Host: www.jackcheng.comUser-Agent: Mozilla/5.0 (X11; U; OpenBSD i386; en-US;

rv:1.5a) Gecko/20031030 Mozilla Firebird/0.6.1Accept: text/css,*/*;q=0.1

Referer: http://www.jackcheng.com/blog/archives/2004/12/ipod_rumors.html

http-graph

• Directed graph history of browsing• Reconstructs graph from referrer and URL

in the header:Referrer Request

• Lets you view your history as you took it• Shows natural “hubs” of information• See also:

http://www.uiweb.com.nyud.net:8090/issues/issue37.htm

Displaying http-graph Output

• Writes a small “dot” file– “dot” part of “graphviz” tool– Use “neato” to graph– Output formats: SVG, PS, PDF, image map– Can make fully interactive!

Example http-graph Output

Grabbing Data with pynids

• tcp.{server, client}.data and just strings• Any string operations will work

– Searchingif “HTTP/1.0” in tcp.client.data:

– Regular Expression searchesif re.search(“HTTP/1.[10]”, tcp.client.data):

– Rewritingstring.replace(req, “GET HTTP/1.0”, “”, 1)

More Fun!• Privacy invasion

– Snarf mail• Log conversations

– IRC, AIM, etc …• Steal files

– FTP, P2P apps, HTTP downloads …• Disrupt sessions

tcp.kill()

New dsniff is written in Python …

flowgrep

• Marries sniffing with regular expressions• A lot like ngrep, tcpkill, and dsniff

– Logs the whole connection, not just a packet• Look for data in streams using regular

expressions• Log or kill selected streams• Dirt cheap IDS or IPS

– Under 400 lines of code

Resources

• http://www.tcpdump.org/• http://www.packetfactory.net/projects/libnids/• http://monkey.org/~provos/libevent/• http://monkey.org/~dugsong/{dpkt, pycap}• http://oss.coresecurity.com/projects/pcapy.html• http://monkey.org/~jose/software/flowgrep/• http://pilcrow.madison.wi.us/pynids/

Additional Resources

• Stevens, TCP/IP Illustrated vols 1 and 2• Schiffman, Building Open Source Network

Security Tools• RFCs from the IETF