IPVS for Docker Containers

Andrey SibiryovBulldozer Operator at Uber

This guy on the stage

2

…who the hell are you and why should we believe a Russian?

What is IPVS…and why didn’t we hear about it before?

International Pig Veterinary Society

4

IP Virtual Server

5

IP Virtual Server

6

…and why didn’t we hear about it before?

IP Virtual Server

7

Works inside the Linux Kernel, based on Netfilter.Supports TCP, SCTP & UDP, v4 and v6.8+ methods: WRR, WLC, LBLCR, SH and much more – plugins.NAT, Tunneling, Direct Routing.Address bundling via FWMark services.Most importantly, it is bloody fucking fast!

IP Virtual Server

8

…and a little bit more about these weird acronyms.

IPIPEncapsulates IP

Routable anywhere

NATRewrites DST IP

Same L4

DSRRewrites DST MAC

Same L2

IP Virtual Server

9

…and a few more words about DR since it’s awesome!

TPS

50

66

82

98

114

130

Single IPVS DR HAProxy MySQL Proxy

I don’t need this

10

…and why would we load balance and route anything at all?

I don’t need this

11

…also, our nginx, haproxy, hipache, vulcand setup works fine – get off the stage, please!

I don’t need this

12

…and we run our stuff in the cloud, it takes care of everything – our work is perpetual siesta!

IP Virtual Server

13

IPVS NGINX HAPROXY AWS ELB

OSI layer L4 L7 L7 L4TCP ✅ ℹ ✅ ✅

UDP ✅ 🚫 🚫 🚫

Dynamic configuration ✅ 🚫 ℹ ✅

Forwarding methods 4 NAT NAT NATBalancing methods 8+ 3 6 RR

Health checks ✅ 🚫 ✅ ✅

What is IPVS…and how do we use it since it sounds so amazing!

GORB

15

Go Routing and Balancing

github.com/kobolog/gorb

What is GORB

16

…and why is it cool for Docker Containers?

17

REST API daemon to control IPVS.Simple GET, PUT, DELETE interface.Docker Link to control GORB.TCP, HTTP health checks built-in.Written in Go, LGPL licensed, on GitHub.Docker + GORB + MacVLAN network plugin + IPVS DR forwarding mode = ❤

What is GORB…and why is it cool for Docker Containers?

What is GORB…and how do we use it? Live demo or GTFO!

What is GORB

19

The guy on the stage is doing roughly this right now.

kobolog@gorb:~$ go install github.com/kobolog/gorb

kobolog@gorb:~$ sudo modprobe ip_vs

kobolog@gorb:~$ sudo gorb &

kobolog@docker:~$ go install github.com/kobolog/gorb/gorb-docker-link

kobolog@docker:~$ export DOCKER_HOST=<...>

kobolog@docker:~$ gorb-docker-link -r gorb:4672 &

kobolog@docker:~$ docker run -d -p 80 nginx

kobolog@gorb:~$ curl -i -XGET http://localhost:80

A few more words about BGP

20

…black belt in networking is not complete without a few words about BGP.

How stable is GORB

21

…and is it production-ready? Can we blame you if it doesn’t work?

«I'll let you in on a secret: my pet hamster did all the coding. I was just a channel, a «front» if you will, in my pet's grand plan. So, don't blame me if there are bugs. Blame the cute, furry one» —Rusty Russel

22

€25,000

23

…is the price for an enterprise hardware load-balancer.

€0

24

…is a price for IPVS and BGP (and GORB).

Works on commodity hardware.No special software or tools, no SNMP.1000-word man page vs. 1000-page vendor manual.1 GBPS line rate = 1% CPU in DR mode.IPVS is around for more than 15 years.

25

A good way to spend €25,000

Thank you!Andrey Sibiryov@kobolog me@kobology.ru