MULTITENANCY IN KUBERNETES
WHAT COMPANIES CARE ABOUT
Velocity
2
Cost
Hello!
I AM KATHARINA PROBST
I’m a Senior Engineering Manager at Google.You can find me at www.linkedin.com/in/katharina.probst
3
WHY MULTITENANCY
4
KUBERNETES AT A GLANCE
5
masteruser CLI/API/UI
node
node
node
NODES
ONE USER, ONE CLUSTER
6
masteruser CLI/API/UI
node
node
node
NODES
MULTIPLE USERS, MULTIPLE CLUSTERS
7
masteruser CLI/API/UI
node
node
node
NODES
masteruser CLI/API/UI
node
node
node
NODES
SPRAWL OF MANY CLUSTERS
8
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
HOW DOES THIS SCALE FINANCIALLY?
9
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
HOW DOES THIS SCALE OPERATIONALLY?
10
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
masteruser CLI/API/UI
kubelet
kubelet
kubelet
NODES
...Namespace 2
Namespace 1
...
MANY USERS, ONE CLUSTER
11
master
User 2
CLI/API/UIUser 3
User 1
User n
Namespace n
MULTITENANCY USERS
12
CLUSTER ADMIN
◦ Set up cluster and namespaces. Set up resource limits.
◦ Ensure consistency across namespaces in the cluster.
◦ Operate the clusters (e.g., respond to incidents).
CLUSTER USER
◦ Start/stop/manage their own app(s) in their own namespace(s).
◦ Understand their namespace(s)’ resource limits.
◦ Don’t trample on other tenants.
NAMESPACE ADMIN
Admin rights to specific namespace(s)
MODELS OF MULTITENANCY
13
HARD MULTITENANCY
◦ Zero-trust tenants
◦ Not yet widely used in production
◦ Ongoing work in kubernetes community to strengthen
14
WHAT WILL HARD MULTITENANCY TAKE?
15
WHAT WILL HARD MULTITENANCY TAKE?
◦ Tenants can’t
▫ DoS/impact access to others’ resources
▫ See each others’ stuff (e.g., by intercepting network traffic or
accessing stored data)
16
WHAT WILL HARD MULTITENANCY TAKE?
◦ Tenants can’t
▫ DoS/impact access to others’ resources
▫ See each others’ stuff (e.g., by intercepting network traffic or
accessing stored data)
◦ Resource objects don’t collide, e.g., custom controllers/CRDs
17
WHAT WILL HARD MULTITENANCY TAKE?
◦ Tenants can’t
▫ DoS/impact access to others’ resources
▫ See each others’ stuff (e.g., by intercepting network traffic or
accessing stored data)
◦ Resource objects don’t collide, e.g., custom controllers/CRDs
◦ Control plane (master) resources are shared fairly
MODELS OF MULTITENANCY
18
SOFT MULTITENANCY
◦ Tenants are more trusted◦ Often used within one enterprise◦ Different teams → different namespaces
◦ Used in practice and in production◦ Often used in combination with infrastructure/platform built on
top by centralized team for consistent security, networking, etc.
MULTITENANCY PRIMITIVES
19
ACCESS CONTROL
Use policies to ensure that tenants can access only what they should have access to
FAIR SHARING
Enforce limits per tenant[Better developed on data plane; control plane WIP]
ISOLATION
Ensure tenants cannot access each others’ workloads, secrets, etc. (security isolation)
ACCESS CONTROL - RBAC
20
ClusterRole A pre-set of capabilities, cluster-wide
Role Like ClusterRole, but namespace-scoped
ClusterRoleBinding Give permissions defined in a ClusterRole
RoleBinding Like ClusterRoleBinding, but namespace-scoped
◦ Use RBAC for controlling access to Secrets etc.
◦ Pod Security Policy (e.g., access to volume types, privileged): enable
fine-grained authorization of pod creation and update
◦ Network Policy (ingress, egress): control which pods can talk to each other
◦ Make Custom Resource Definitions namespace-scoped
◦ Sandboxes, e.g., gVisor
▫ Ensure security isolation of pods
▫ Ensure that information is not leaked between untrusted tenants
21
ISOLATION
◦ Resource Quotas (with Limit Range defaults) for CPU, memory, object
counts
◦ Pod Priority, Quality of Service Classes
◦ Node
▫ Taints & Tolerations: allow a node to repel a set of pods
▫ Labels and node selectors (less flexible than Affinity)
◦ Pod
▫ Affinity
▫ Pod Anti-affinity
22
FAIR SHARING - DATA PLANE (NODES)
23
DATA PLANE VS. CONTROL PLANE MULTITENANCY
Much of what we’ve talked about is sharing the data plane (nodes)
24
CONTROL PLANE MULTITENANCY
...Namespace 2
Namespace 1
...
MANY USERS, ONE CLUSTER
25
master
User 2
CLI/API/UIUser 3
User 1
User n
Namespace n
Not able to manage multiple clusters →
1:1 mapping between master and cluster, though cluster may have multiple
tenants
26
MULTITENANT API SERVER?
...Namespace 2
Namespace 1
...
MANY USERS, ONE CLUSTER
27
master
User 2
CLI/API/UIUser 3
User 1
User n
Namespace n
All tenants share master (incl. Secrets, ConfigMap), but RBAC helps
Little protection against individual tenants DoSing each other
28
MULTITENANT API SERVER?
Max inflight requests: mechanism for protecting API server against CPU and
memory overloads
Current problem to address: Tenants can crowd each other out (accidentally or
on purpose)
29
BETTER API SERVER FAIRNESS (UNDERWAY)
Proposal generalizes … max-in-flight request handler in the apiserver to make
more distinctions among requests and provide prioritization and fairness among
the categories of requests.”
30
BETTER API SERVER FAIRNESS (UNDERWAY)
BETTER API SERVER FAIRNESS (UNDERWAY)
Multiple priority levels, each has queues
Within each priority level, queues compete evenly
31
WHAT COMPANIES CARE ABOUT
Velocity
32
Cost
◦ Use multitenancy for improved resource efficiency, cost, and operations
◦ Different models of multitenancy
▫ Soft
▫ Hard
◦ Hard multitenancy is still work in progress
◦ Soft multitenancy is already used in production by various companies
▫ But is often coupled with a shared internal platform to gain
consistency of networking, security, etc. across teams
33
KEY TAKE-AWAYS
QUESTIONS?
◦ Project plan for multitenancy
▫ Building CRD for more automatic management of namespaces
◦ API Machinery KEP for improved resource sharing
◦ Cluster multi-tenancy for a good overview
35
Links for additional details
Top Related