SHARKFEST ‘10 | Stanford University | June 14–17, 2010 Wireshark in the Large Enterprise June...

SHARKFEST ‘10 | Stanford University | June 14–17, 2010

Wireshark in the Large EnterpriseJune 16, 2010

Hansang BaeSenior Vice President | Citi (f.k.a. Citigroup)Email: [email protected] REFER TO THE “ANSWERSHEET.DOCX” FILE FOR ADDITIONAL INFORMATION ABOUT THIS PRESENTATION.THESE SESSIONS WILL BE AVAILABLE ON YOUTUBE: HTTP://WWW.YOUTUBE.COM/USER/HANSANGB

SHARKFEST ‘10Stanford UniversityJune 14-17, 2010

Microsoft Office Word Document


Please Let TCP Do Its Job.

Problem: Application developers escalate an issue with slow file (MQ) transfers.

Troubleshooting Steps:1. What should you rule out immediately?2. What affects throughput and why?3. Look for patterns and ask the right questions. Quick examination

would reveal what? Doesn’t it look normal? Can you spot the issue quickly? Were you guys paying attention yesterday?!?

4. Use the graphing tools. Picture is worth a thousand words.5. Setup your Wireshark environment in a standard way. Use

Configuration Manager to help you.

B2BXfer.pcap


Don’t Jump to Conclusions!

Another application development team escalates a “slowness” problem.

Troubleshooting Steps:1. Trust But Verify (tcp.analysis.flags)2. Look for telltale signs of problems. 3. Who’s sending and who’s receiving? Besides looking at

the name of the file….can you figure it out?4. Apply Occam’s Razor when solving problems.

SendSideFinal.pcapRcvSideFinal.pcap


Another (unusual) Hidden Danger!

Application testing with an external vendor doesn’t work. It tested fine when tested with intra-resources.

Troubleshooting Steps:1. If it works internally but not with an external vendor (reachable via

Internet) what device should you suspect? Learn to Divide and Conquer – the power of binary search!

2. Have “High Bandwidth Conversations” with qualified peers.3. Look out for “Defaults” HSB’ism: Defaults are the guardian angels

for the clueless! 4. Another case of “picture is worth a thousand words”

GryphonFinal.pcap


Odd Numbers are Evil? Really?

Software Update System is slow in delivering packages to staging servers. It impacts 300,000+ users!

Troubleshooting Steps:1. Usual Suspects (Duplex, Window size, Pkt loss, and LFN)2. Use the information in the trace to eliminate some of the “usual

suspects.” Not all inefficiencies come into play. Does Window come into play here?

3. Do I need to see the SYN/SYN+ACK to see what environment this is? What other options are there?

4. Use Time Reference markings liberally? 5. Case of “too much of a good thing”

DMFinal.pcap LMFinal.pcap


Another Zebra Case!

Users are calling into the helpdesk because the Citrix sessions are dying.

Main Concept:1. Applications traversing the Internet play by a different set of

rules/standards. Packet loss is a way of life.2. Do you **REALLY** know TCP?3. Did you pick up on why the 500ms delay is significant? 4. What is Fast Retransmit and how is it different from “regular”

Retransmission?5. Learn the art of spotting something unusual. But first, you need to

understand “what’s unusual.”

HUDFinal.pcap


Wan Optimization

After upgrading WAN optimization appliances, tellers started reporting intermittent printing issues. Transient problems like these are the toughest to resolve. What was the time to Resolution? Three days - thanks to packet captures.

Main Concept:1. Last change was OS upgrade on the wan optimization appliance, so

start there.2. Capturing in the right capture points is critical. Why?3. Is it worth looking at TCP Session #2? 4. What should you compare? What can you compare?5. Sake Blok’s session last year on SSL decryption was VERY helpful!

ServerFinal.pcap ClientFinal.pcap


Wan Optimization (Con’t)

SHARKFEST ‘10 | Stanford University | June 14–17, 2010 Wireshark in the Large Enterprise June...

Documents

Transcript of SHARKFEST ‘10 | Stanford University | June 14–17, 2010 Wireshark in the Large Enterprise June...