Using NetLogger and Web100 for TCP analysis
description
Transcript of Using NetLogger and Web100 for TCP analysis
![Page 1: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/1.jpg)
Using NetLogger and Web100 for TCP analysis
Data Intensive Distributed Computing Group Lawrence Berkeley National Laboratory
Brian L. Tierney
![Page 2: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/2.jpg)
The Problem
• The Problem:– TCP throughput on very high-speed networks is
often disappointing.• Why is this? What is the cause?• Using tuned TCP buffers, txqueuelen, and see no loss, but
performance is still poor. Why!? – Want to test a modification to TCP (eg.: HS-TCP,
Fast TCP,etc)• What are the effects of this modification?
• The Solution– Instrumented TCP and analysis tools
![Page 3: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/3.jpg)
Web100 + NetLogger
• Web100 (PSC + NCAR) provides– Ability to instrument TCP stack in detail
• NetLogger (LBNL) provides– Ability to correlate data from varies sources based on time– Easy way to collect data from multiple clients/servers reliably– Visualization and analysis tools
![Page 4: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/4.jpg)
Important Web100 Variables for understanding TCP
• TCP throughput directly related to the Congestion Window size (CWND)• The following may restrict/reduce CWND
– CongestionSignals (includes Retransmits, FastRetransmits, & ECN)– MaxRwinRcvd: receiver advertised maximum– SendStall: Interface queue is full (txqueuelen)– X_OtherReductionsCV: TCP Congestion Window Validation
(RFC2861). Reduce CWND when the actual window is smaller than CWND for more than 1 RTT
– X_OtherReductionsCM: Linux “CWND Moderation” (explained below)• These variables indicate if the throughput is limited by the sender, the
receiver, or the network– SndLimTimeRwin – SndLimTimeCwnd – SndLimTimeSender
![Page 5: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/5.jpg)
Net100 pyWAD
• WAD = Work Around Daemon– pyWAD: python version implemented by Jason Lee, LBNL
• Originally conceived as a tuning daemon– E.g: auto-tune TCP buffer size, etc.– Can also be used for transparent instrumentation, and can generate
derived events• Sample Configuration file
[monitor iperf_client]src_addr: 0.0.0.0 # all source addresses src_port: 0 # any source portdst_addr: 0.0.0.0 # any destination addressdst_port: 5005 # all traffic on port 5555 [NetLogger]web100.CongestionSignals: CongestionSignalsweb100.SendStall: SendStallweb100.CurCwnd: CurCwndweb100.SmoothedRTT: SmoothedRTTweb100.OtherReductions: OtherReductionsAveBW1: (DataBytesOut*8)/(SndLimTimeRwin + SndLimTimeCwnd + SndLimTimeSender)[PyWAD]outputdest: file:///tmp/iperf.test.2.logpolltime: 0.5
![Page 6: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/6.jpg)
“Normal” Plot: Standard TCP
![Page 7: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/7.jpg)
SC02 Test Environment
LBL test host1.4 GHz
NERSC test host2 x 1 Ghz
ANL test host1.13 GHz
SC02 test host2 x 1.4 GHz
NIKHEF test host2.4 GHz
900 Mbps
580 Mbps
900 Mbps
780 Mbps
Network speed = Measured UDP throughput
![Page 8: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/8.jpg)
With Net100 Mods: HS-TCP + IFQ
Amsterdam to SC02
![Page 9: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/9.jpg)
Uneven Parallel Streams
Amsterdam to LBNLNote variation of smoothedRTT varies on slow stream
![Page 10: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/10.jpg)
OtherReductions reduce CWND
Amsterdam to SC02HS-TCP turned on
![Page 11: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/11.jpg)
Coloration of Sack and OtherReductionsCM
CWND drops
SACKs
OtherReductionsCM
![Page 12: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/12.jpg)
Linux OtherReductionsCM Code
/* CWND moderation, preventing bursts due to too big ACKs in dubious situations. */
static __inline__ void tcp_moderate_cwnd(struct tcp_opt *tp){ tp->snd_cwnd = min(tp->snd_cwnd, tcp_packets_in_flight(tp)+tcp_max_burst(tp)); tp->snd_cwnd_stamp = tcp_time_stamp; }
/* Slow start with delack produces 3 packets of burst */static __inline__ __u32 tcp_max_burst(struct tcp_opt *tp){ return 3; }
/* This determines how many packets are "in the network" to the best of our knowledge. Read this equation as:
* "Packets sent once on transmission queue" MINUS * "Packets left network, but not honestly ACKed yet" PLUS * "Packets fast retransmitted" */static __inline__ unsigned int tcp_packets_in_flight(struct tcp_opt *tp){ return tp->packets_out - tp->left_out + tp->retrans_out;}
![Page 13: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/13.jpg)
Linux TCP Bug
Path = Amsterdam to LBLThis happens when CWND gets too large
![Page 14: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/14.jpg)
Conclusions and Recommendations
• Web100 + NetLogger provide a very useful method for analyzing Linux TCP behavior
• Parallel streams may be a bad idea with well tuned streams
• Personal Recommendation:– All Linux-based TCP testing be based on the
Web100 kernel, and always run pyWAD to collect TCP instrumentation data during all tests
– This will can always help answer the question: “Why did that happen?”
![Page 15: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/15.jpg)
For More Information
• Web100: http://www.web100.org/• NetLogger: http://www-didc.lbl.gov/NetLogger/• pyWAD: http://www-didc.lbl.gov/net100/pyWAD.html
• Email: [email protected]
![Page 16: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/16.jpg)
Extra Slides
![Page 17: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/17.jpg)
Summary Results
• Things to note:– TCP was typically 5 times slower than UDP– Parallel streams VERY uneven on paths 1 and 2– Parallel streams slower than single stream on path 1– SendStalls were only seen on paths 1 and 2, so net100 IFQ
setting will only effect these paths – Floyd High-Speed TCP helped on paths 3 and 4– Large standard deviation on all measurements
Net100 Tuned TCPstandard TCP (Mbps) (FloydAIMD = IFQ = 1)
UDP 1 stream 3 streams 1 stream 3 StreamsAmsterdam to SC02 900 156 83+26+13=122 164 85+25+32=142Berkeley to SC02 780 120 212+111+32=355 250 162+30+14=206Oakload to SC02 580 30 22+22+22=66 64 63+50+37=150Chicago to SC02 900 140 72+48+46=166 161 79+77+46=202
![Page 18: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/18.jpg)
SendStalls Reducing CWND
Amsterdam to SC02; HS-TCP
![Page 19: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/19.jpg)
Bursty Sender
Oakland to SC02Send bursts due to large txqueuelen on send host
![Page 20: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/20.jpg)
Uneven Parallel Streams
Amsterdam to SC02Note variation of smoothedRTT varies on different streams
![Page 21: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/21.jpg)
Zoom on Slow Start
ANL to SC02
![Page 22: Using NetLogger and Web100 for TCP analysis](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dc7550346895dcbf48e/html5/thumbnails/22.jpg)
Zoom on Parallel Streams
LBL to SC02