Redis Networking Nerd Down: For Lovers of Packets and Jumbo Frames- John Bullard, Distil Networks
42
For Lovers of Packets and Jumbo Frames Redis Networking Nerd Down
-
Upload
redis-labs -
Category
Technology
-
view
672 -
download
2
Transcript of Redis Networking Nerd Down: For Lovers of Packets and Jumbo Frames- John Bullard, Distil Networks
For Lovers of Packets and Jumbo FramesRedis Networking Nerd Down
Intro
John BullardVP Engineering
Benji TaylorDevOps Lead
We Block BotsDistil
How the Distil Bot Detection Solution Works
As web traffic passes through Distil, the system
1. Fingerprints each incoming connection and compares it to our Known Violators Database
2. If it’s a new fingerprint, validates the browser to determine if it’s a Bot or Not
3. Based on your preferences, automatically tags, challenges, or blocks the bot
Bad Bots Cause Problems
Fortune 500 & Alexa Global 10,000 CustomersEcommerce
Travel
Publishers
Directories
Traditional Media
Marketplace
Services
We Block BotsDistil Architecture
Distil Architecture
Topology - Single Master
Topology - Master : Master
Topology - Master : Slave
Every Packet MattersCustomer Story
Customer Story
○Large Online MarketplaceHigh traffic volume Approaching seasonal peak - 3x trafficAWS EC2 Classic
○ Soft 100k PPS Limit○ Customer concern
Customer Story
○Scaling Up + OutMaster : Slave Topology
○ 9 Node: 1 Master, 8 SlavesC3.8xlarge Instances
○ 36 vCPU, 60 GiB, 10 GigabitProof of Concept
○ Onboard <5% of traffic
POC - Slave Measurements
POC - Master Measurements
POC - Master Measurements
Customer Story
Customer Story
Customer Story
○Bottleneck - Master PPSRedis Master : Write + Replicate
○ All nodes write○ All writes replicate to all nodes
○RequirementsMaster PPS < 100kNo partitioningEC2 Classic$0 Cost Increase
We Block BotsPackets
Packets
Packets
Packets
Packets
Overview
○Jumbo CaveatsNot supported everywhere (Virtualbox)“Jumbo” differs among vendorsEvery component on path must support itSwitches - drop jumbo framesRouters - must support ingress and egress
Overview
○ Tuning - Jumbo frames - AWS Support for jumbo frames:
○ Compute optimized: C3, C4, CC2 ○ General purpose: M3, M4, T2 - GPU: CG1, G2 ○ Memory optimized: CR1, R3 ○ Storage optimized: D2, HI1, HS1, I2
MTU 9001 bytes
Packets
Packets
Packets
Packets
Packets
○TCP Small Packet ProblemApplication sending 1 byte packetsTCP packets have a 40 byte header
○ 20 bytes for TCP○ 20 bytes for IPv4
41 byte packet ○ For 1 byte of useful information
Packets
○Nagle’s algorithm (RFC 896)Combine small outgoing messages
Reduce # of packetsImprove TCP/IP efficiency Trade-off: increased latency
○ Problematic for real-time systems○ Per-socket: TCP_NODELAY
Packets
# Disable TCP_NODELAY on the slave socket after SYNC?## If you select "yes" Redis will use a smaller number of TCP packets and# less bandwidth to send data to slaves. But this can add a delay for# the data to appear on the slave side, up to 40 milliseconds with# Linux kernels using a default configuration.## If you select "no" the delay for data to appear on the slave side will# be reduced but more bandwidth will be used for replication.## By default we optimize for low latency, but in very high traffic conditions# or when the master and slaves are many hops away, turning this to "yes" may# be a good idea.repl-disable-tcp-nodelay no
Packets
Packets
Packets
Master/Slave ChainsCommon Architecture for Read Intensive
ScenariosBalance Write Duplication to Read-SlavesShift PPS to Read-SlavesConfiguration Change - No DevNo Added Cost
○ 1 Master 2 Read-Slaves 6 Edgenodes
Customer Story
Customer Story
Packets
○ApproachEthernet tuning - Enable Jumbo PacketsTCP/IP tuning - Enable “Naggling”Topology - Read-Slave fan-out
Demo
