Introducing WordPress Multitenancy (Wordcamp Vegas/Orlando 2015/WPCampus)
The Docker Multitenancy Problem: A Journey through Infrastructure Hell
-
Upload
peter-klipfel -
Category
Data & Analytics
-
view
121 -
download
2
Transcript of The Docker Multitenancy Problem: A Journey through Infrastructure Hell
![Page 1: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/1.jpg)
A JOURNEY THROUGH INFRASTRUCTURE HELL
MULTITENANCY WITH DOCKER
Peter Klipfel
![Page 2: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/2.jpg)
STOP ME IF YOU WANT TO LOOK AROUND
THERE’S A LOT TO SEE
![Page 3: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/3.jpg)
I WILL PAUSEWHEN I SEE A TURTLE
![Page 4: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/4.jpg)
SOME CONTEXT
![Page 5: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/5.jpg)
WHAT WE’RE TRYING TO DO
EACH USER GETS
▸ Private data storage
▸ Notebook (executable code on our servers)
▸ Deployed microservices
![Page 6: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/6.jpg)
WHAT WE’RE TRYING TO DO
WE NEED
▸ Scalability
▸ Fault Tolerance
▸ Security
![Page 7: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/7.jpg)
HOW HARD IS IT TO CREATE A MULTI-TENANT ELASTICSEARCH CLUSTER?
LET’S START WITH A QUESTION
![Page 8: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/8.jpg)
VERY HARD
![Page 9: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/9.jpg)
MULTITENANT ELASTICSEARCH
POSSIBLE SOLUTIONS
▸ Built in multi tenancy
▸ Shield
▸ Search-guard
![Page 10: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/10.jpg)
MULTITENANT ELASTICSEARCH
NONE OF THEM WORK
▸ Built in multi tenancy: update yml file every user ->
restart
▸ Shield: Not Free
▸ Search-guard: SSL was painful
![Page 11: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/11.jpg)
![Page 12: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/12.jpg)
HOW CAN WE DO THAT?
EACH USER GETS THEIR OWN DATABASE
![Page 13: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/13.jpg)
ELASTICSEARCH INSTANCE PER USER
POSSIBLE SOLUTIONS
▸ Use hosted ES: Really expensive
▸ Use a cloud provider: expensive
▸ Use Docker: not as expensive
![Page 14: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/14.jpg)
DOCKER TO THE RESCUE!
![Page 15: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/15.jpg)
HOW DO WE CREATE DOCKER CONTAINERS ON DEMAND?
BUT WAIT
![Page 16: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/16.jpg)
DOCKER CONTAINERS ON DEMAND
POSSIBLE SOLUTIONS
▸ Mesos (+ marathon)
▸ Docker Swarm
▸ Kubernetes
![Page 17: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/17.jpg)
DOCKER CONTAINERS ON DEMAND
WHAT ARE THOSE TOOLS?
▸ Container schedulers
▸ APIs to run a docker container somewhere in the
cluster
▸ Uniform cluster nodes
![Page 18: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/18.jpg)
DOCKER CONTAINERS ON DEMAND
WHAT ARE THOSE TOOLS?
MASTER
AGENT
AGENT
AGENT
AGENT
AGENT
CONTAINER
CONTAINER
CONTAINER
![Page 19: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/19.jpg)
![Page 20: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/20.jpg)
DOCKER CONTAINERS ON DEMAND
THE PROBLEMS
▸ How do users get to their services (databases)?
▸ What if a node goes down?
▸ How do I separate users?
![Page 21: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/21.jpg)
HOW DO USERS GET TO THEIR DATABASES?
![Page 22: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/22.jpg)
SERVICE ACCESS
WHAT ARE THOSE TOOLS?MAST
ER
AGENT
CONTAINER
CONTAINER
CONTAINER
AGENT
CONTAINER
CONTAINER
CONTAINER
AGENT
CONTAINER
CONTAINER
CONTAINER
AGENT
CONTAINER
CONTAINER
CONTAINER
REVERSE
PROXY
![Page 23: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/23.jpg)
SERVICE ACCESS
REVERSE PROXY
▸ Nginx (reloads good)
▸ HAProxy (reloads bad)
▸ And we will need Consul
![Page 24: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/24.jpg)
SERVICE ACCESS
CONSUL: THE EASIEST WAY
▸ We need Registrator on every node
▸ consul-dns creates routing
▸ consul-template builds nginx config
![Page 25: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/25.jpg)
SERVICE ACCESS
NOW OUR REVERSE PROXY WORKS!MAST
ER
AGENT
CONTAINER
CONTAINER
CONTAINER
AGENT
CONTAINER
CONTAINER
CONTAINER
AGENT
CONTAINER
CONTAINER
CONTAINER
AGENT
CONTAINER
CONTAINER
CONTAINER
REVERSE
PROXY
…
![Page 26: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/26.jpg)
![Page 27: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/27.jpg)
SERVICE ACCESS
POTENTIAL ALTERNATIVES
▸ ETCD
▸ MesosDNS
▸ Zookeeper
![Page 28: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/28.jpg)
WHAT IF A NODE GOES DOWN?
GREAT! USERS CAN ACCESS THINGS!
![Page 29: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/29.jpg)
STATEFUL SERVICES
PROBLEMS
▸ Containers have different fs mounts on each instance
▸ Node spin-up is non-deterministic (which disk will it
use)
▸ Network file systems require implementation
changes
![Page 30: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/30.jpg)
STATEFUL SERVICES
SOME SOLUTIONS
▸ We can mount docker container filesystems with
volumes
▸ Can specify certain nodes for services
▸ Force stateful services to same node
![Page 31: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/31.jpg)
CLUSTERINGSOLUTION:
![Page 32: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/32.jpg)
STATEFUL SERVICES
CLUSTERING
▸ Failure is ok, as long as it’s not the whole cluster
▸ Storage can be ephemeral
▸ Most databases cluster
![Page 33: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/33.jpg)
![Page 34: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/34.jpg)
HOW DO WE KEEP OUR USERS SEPARATED?
GREAT! LET’S CLUSTER
![Page 35: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/35.jpg)
NETWORK ISOLATION
THEY’RE ALL ON THE SAME SYSTEMMAST
ER
AGENT
CONTAINER
CONTAINER
CONTAINER
AGENT
CONTAINER
CONTAINER
CONTAINER
AGENT
CONTAINER
CONTAINER
CONTAINER
AGENT
CONTAINER
CONTAINER
CONTAINER
REVERSE
PROXY
…
![Page 36: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/36.jpg)
NETWORK ISOLATION
PROBLEMS WITH CLUSTERING
▸ Reverse proxy works only for HTTP
▸ Don’t want to DOS the internal network
▸ Need isolation between users
![Page 37: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/37.jpg)
NETWORK ISOLATION
SOLUTION: DOCKER OVERLAY NETWORK▸ Weave
▸ Callico
▸ Flannel
![Page 38: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/38.jpg)
WE JUST REINVENTED OPENSTACK…
BUT WAIT
![Page 39: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/39.jpg)
![Page 40: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/40.jpg)
![Page 41: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/41.jpg)
NETWORK ISOLATION
PROBLEMS WITH OPENSTACK
▸ Maintaining it sucks
▸ Upgrading it sucks
▸ Paying for it sucks
![Page 42: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/42.jpg)
SO I USED OPENSTACK FOR A WHILE
![Page 43: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/43.jpg)
NETWORK ISOLATION
HOW IT WORKED
▸ User gets their own account
▸ Every user gets their own network
▸ Every user gets their own persistent storage
![Page 44: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/44.jpg)
![Page 45: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/45.jpg)
KUBERNETES
AND AFTER IT STOPPED SCALING I TRIED
![Page 46: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/46.jpg)
GOOGLE CONTAINER ENGINE
AND AFTER IT STOPPED SCALING I TRIED
![Page 47: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/47.jpg)
GOOGLE CONTAINER ENGINE (GKE)
THE BEST SOLUTION I HAVE FOUND
▸ Persistent volumes
▸ Decent library support
▸ Hopeful networking promised land
![Page 48: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/48.jpg)
GOOGLE CONTAINER ENGINE (GKE)
PERSISTENT VOLUMES
▸ I don’t need automated clustering if disks are
persistent
▸ Manual deploy for customers that require larger
clusters
▸ Can separate disk utilization by service
![Page 49: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/49.jpg)
GOOGLE CONTAINER ENGINE (GKE)
HOPEFUL NETWORKING PROMISED LAND▸ Configuration defines subnetwork id
▸ Subnets can exist across data centers
▸ Lots of opportunities for more clever reverse
proxying
![Page 50: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/50.jpg)
CONCLUSIONWHAT HAVE WE LEARNED?
![Page 51: The Docker Multitenancy Problem: A Journey through Infrastructure Hell](https://reader035.fdocuments.in/reader035/viewer/2022081605/58a1495d1a28abf8068b691b/html5/thumbnails/51.jpg)
CONCLUSION
WHAT HAVE WE LEARNED?
▸ Docker is a glorified package manager
▸ Complex microservice architectures are still hard
▸ The promised land is close