Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency...
Transcript of Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency...
![Page 2: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/2.jpg)
Outline
• TCP Fast Open allows to reduce latency and significantly improve user-experience
• However, naive firewalls and bad Intrusion Detection Systems got in our way
We should change that!
![Page 3: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/3.jpg)
Latency matters
[1] “More Bandwidth Doesn’t Matter (much)”. M. Belshe. 2010 (https://goo.gl/X8rE6Q).
PLT of 25 most popular websites (Latency = 60ms)
Page
Loa
d Ti
me
[ms]
750
4000
Bandwidth [Mbps]1 2 3 4 5 6 7 8 9 10
![Page 4: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/4.jpg)
Latency matters
[1] “More Bandwidth Doesn’t Matter (much)”. M. Belshe. 2010 (https://goo.gl/X8rE6Q).
PLT of 25 most popular websites (Latency = 60ms)
Page
Loa
d Ti
me
[ms]
750
4000
Bandwidth [Mbps]1 2 3 4 5 6 7 8 9 10
PLT of 25 most popular websites (Bandwidth = 5Mbps)
Page
Loa
d Ti
me
[ms]
750
4000
Latency [ms]240 220 200 180 160 140 120 100 80 60 40 20 0
![Page 5: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/5.jpg)
Latency matters
• [2] measured impact of latency on service revenue
• Direct correlation between latency and revenue:
‣ 100ms of additional delay has significant impact on the revenue and customer satisfaction
[2] “Measuring and Mitigating Web Performance Bottlenecks in Broadband Access Networks”. S. Sundaresan, et al. ACM IMC 2013.
![Page 6: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/6.jpg)
Transmission Control Protocol
• Used for 95% of the Internet’s traffic
• Provides a reliable and in-order byte-stream service
• 3-way handshake to establish the connection
![Page 7: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/7.jpg)
The TCP joke
• “Hi, I’d like to hear a TCP joke.”
• “Hello, would you like to hear a TCP joke?”
• “Yes, I’d like to hear a TCP joke.”
• “Ok, I’ll tell you a TCP joke.”
• …
![Page 8: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/8.jpg)
TCP Handshake is expensive
SYN
SYN/ACK
Data
ACKData
Client Server
![Page 9: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/9.jpg)
TCP Handshake is expensive
SYN
SYN/ACK
Data
ACKData
Client Server
1 RTT before any data is sent
Cellular Network’s RTT can range in the
hundreds of milliseconds
![Page 10: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/10.jpg)
TCP Fast Open (RFC 7413)
Accelerating the TCP Handshake
![Page 11: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/11.jpg)
TCP Fast Open (TFO)
• Allows clients to send SYN with data
• Enables servers to reply right away with the response
• Protects itself against DoS through a cookie, unique for each client-IP
• Standardized at the IETF - RFC 7413
[3] “TCP Fast Open”. Y. Cheng, J. Chu, S. Radhakrishnan, A. Jain. IETF RFC 7413. 2014
![Page 12: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/12.jpg)
TFO at Apple• TCP Fast Open in iOS 9 and OS X 10.11 (and later)
• Used for an Apple Service on all iOS and OS X devices
• Public API by using connectx(2)
• Overall, very beneficial
But, Firewalls got in our way
![Page 13: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/13.jpg)
TFO in details1. Cookie Exchange
SYN TFO Cookie Request
SYN/ACK
TFO Cookie = X
Client Server
Generate Cookie: X = Hash(Client_IP, secret)
![Page 14: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/14.jpg)
TFO in details2. Sending SYN + data
SYN + data TFO Cookie = X
SYN/ACK
Client Server
Verify Cookie: X == Hash(Client_IP, secret)
![Page 15: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/15.jpg)
TFO in details3. Server replies with data
SYN + data TFO Cookie = X
SYN/ACK
Client Server
Verify Cookie: X == Hash(Client_IP, secret)
Data
ACK
![Page 16: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/16.jpg)
Middlebox issues with TCP Fast Open
… and their negative impact
![Page 17: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/17.jpg)
Middlebox issues• Bad Middleboxes and Firewalls respond badly to
TCP Fast Open
‣ Suppress TCP options
‣ Drop packets
‣ Mark entire connection as “invalid”
‣ Blackhole the clients
![Page 18: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/18.jpg)
Using a new TCP optionIssue
Simplistic middleboxes remove unknown TCP options
![Page 19: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/19.jpg)
Using a new TCP optionIssue
Simplistic middleboxes remove unknown TCP options
SYN TFO Cookie Request
SYN/ACK
Client Server
SYN
![Page 20: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/20.jpg)
Using a new TCP optionIssue
Simplistic middleboxes remove unknown TCP options
Impact
Clients cannot use TFO, and thus pay a latency cost compared to well-behaving networks
![Page 21: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/21.jpg)
Using a new TCP optionIssue
Simplistic middleboxes drop segments with unknown TCP options
![Page 22: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/22.jpg)
Using a new TCP optionIssue
Simplistic middleboxes drop segments with unknown TCP options
SYN TFO Cookie Request
SYN
Client Server
RTO (1s)
![Page 23: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/23.jpg)
Using a new TCP optionIssue
Simplistic middleboxes drop segments with unknown TCP options
Impact
Client has to retransmit the SYN-segment without the TCP option. The user experiences a high page-
load-time.
![Page 24: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/24.jpg)
Sending SYN+dataIssue
Naive middleboxes drop SYN segments with data
![Page 25: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/25.jpg)
Sending SYN+dataIssue
Naive middleboxes drop SYN segments with data
SYN + data TFO Cookie = X
SYN
Client Server
RTO (1s)
![Page 26: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/26.jpg)
Sending SYN+dataIssue
Naive middleboxes drop SYN segments with data
Impact
Clients has to retransmit the SYN-segment without the TCP option. The user experiences a high page-
load-time.
![Page 27: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/27.jpg)
Acknowledging SYN+dataIssue
The server acknowledges the SYN+data, thus more than the initial sequence number. Middleboxes might
drop the SYN/ACK.
![Page 28: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/28.jpg)
Acknowledging SYN+dataIssue
The server acknowledges the SYN+data, thus more than the initial sequence number. Middleboxes might
drop the SYN/ACK.
SYN/ACK (Ack=11)
Client ServerSYN (Seq=0, len=10) + data
Expected Ack=1SYN/ACK (Ack=11)
![Page 29: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/29.jpg)
Acknowledging SYN+dataIssue
The server acknowledges the SYN+data, thus more than the initial sequence number. Middleboxes might
drop the SYN/ACK.
Impact
The middlebox keeps on blocking the server’s SYN/ACK. The session never becomes established.
![Page 30: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/30.jpg)
Server sends data right before 3-way handshake completes
Issue
Bad Intrusion Detection Systems (IDS) start blackholing the client
![Page 31: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/31.jpg)
Server sends data right before 3-way handshake completes
Issue
Bad Intrusion Detection Systems (IDS) start blackholing the client
SYN + data TFO Cookie = X
Client Server
SYN/ACKData
ACK
![Page 32: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/32.jpg)
Server sends data right before 3-way handshake completes
Issue
Bad Intrusion Detection Systems (IDS) start blackholing the client
Impact
Client loses connectivity to the server. Subsequent connections (non-TFO) also might be blocked by the
IDS.
![Page 33: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/33.jpg)
How common is this?
Mostly, TFO works successfully (~80% success-rate).
But…
![Page 34: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/34.jpg)
How common is this?
Mostly, TFO works successfully (~80% success-rate).
But…
100% of the users of the affected networks are penalized
![Page 35: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/35.jpg)
Conclusion
• Latency has a direct impact on user-experience
• TCP Fast Open allows to significantly reduce latency
• Bad middleboxes are interfering with TCP Fast Open
Vendors and operators:Take TFO into account for a better user-experience
![Page 36: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/36.jpg)
References
[1] “More Bandwidth Doesn’t Matter (much)”. M. Belshe. 2010 (https://goo.gl/X8rE6Q). [2] “Measuring and Mitigating Web Performance Bottlenecks in Broadband Access Networks”. S. Sundaresan, et al. ACM IMC 2013. [3] “TCP Fast Open”. Y. Cheng, J. Chu, S. Radhakrishnan, A. Jain. IETF RFC 7413. 2014
![Page 37: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/37.jpg)
Backup-slides
![Page 38: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/38.jpg)
TFO and idempotency• Data sent in a SYN might reach the server twice
SYN + data
SYN/ACK
ACK
Client Server
SYN + data
SYN + data
SYN + data got delayed in the
network
TCP-session terminates
Old SYN + data finally reaches the
server
![Page 39: Network support for TCP Fast Open - NANOG ArchiveLatency matters • [2] measured impact of latency on service revenue • Direct correlation between latency and revenue: ‣ 100ms](https://reader034.fdocuments.in/reader034/viewer/2022042414/5f2f28456cf38678625258f0/html5/thumbnails/39.jpg)
TFO and idempotency
• Use TFO only with “idempotent” data (aka., data that can be received twice by the server)
E.g.,:
• TLS (ClientHello)
• HTTP-Requests