IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost,...

MULTITENANCY IN KUBERNETES

WHAT COMPANIES CARE ABOUT

Velocity

Hello!

I AM KATHARINA PROBST

I’m a Senior Engineering Manager at Google.You can find me at www.linkedin.com/in/katharina.probst

WHY MULTITENANCY

KUBERNETES AT A GLANCE

masteruser CLI/API/UI

ONE USER, ONE CLUSTER

MULTIPLE USERS, MULTIPLE CLUSTERS

SPRAWL OF MANY CLUSTERS

kubelet

HOW DOES THIS SCALE FINANCIALLY?

kubelet

HOW DOES THIS SCALE OPERATIONALLY?

kubelet

...Namespace 2

Namespace 1

MANY USERS, ONE CLUSTER

master

User 2

CLI/API/UIUser 3

User 1

User n

Namespace n

MULTITENANCY USERS

CLUSTER ADMIN

◦ Set up cluster and namespaces. Set up resource limits.

◦ Ensure consistency across namespaces in the cluster.

◦ Operate the clusters (e.g., respond to incidents).

CLUSTER USER

◦ Start/stop/manage their own app(s) in their own namespace(s).

◦ Understand their namespace(s)’ resource limits.

◦ Don’t trample on other tenants.

NAMESPACE ADMIN

Admin rights to specific namespace(s)

MODELS OF MULTITENANCY

HARD MULTITENANCY

◦ Zero-trust tenants

◦ Not yet widely used in production

◦ Ongoing work in kubernetes community to strengthen

WHAT WILL HARD MULTITENANCY TAKE?

◦ Tenants can’t

▫ DoS/impact access to others’ resources

▫ See each others’ stuff (e.g., by intercepting network traffic or

accessing stored data)

◦ Tenants can’t

◦ Resource objects don’t collide, e.g., custom controllers/CRDs

◦ Tenants can’t

◦ Resource objects don’t collide, e.g., custom controllers/CRDs

◦ Control plane (master) resources are shared fairly

MODELS OF MULTITENANCY

SOFT MULTITENANCY

◦ Tenants are more trusted◦ Often used within one enterprise◦ Different teams → different namespaces

◦ Used in practice and in production◦ Often used in combination with infrastructure/platform built on

top by centralized team for consistent security, networking, etc.

MULTITENANCY PRIMITIVES

ACCESS CONTROL

Use policies to ensure that tenants can access only what they should have access to

FAIR SHARING

Enforce limits per tenant[Better developed on data plane; control plane WIP]

ISOLATION

Ensure tenants cannot access each others’ workloads, secrets, etc. (security isolation)

ACCESS CONTROL - RBAC

ClusterRole A pre-set of capabilities, cluster-wide

Role Like ClusterRole, but namespace-scoped

ClusterRoleBinding Give permissions defined in a ClusterRole

RoleBinding Like ClusterRoleBinding, but namespace-scoped

◦ Use RBAC for controlling access to Secrets etc.

◦ Pod Security Policy (e.g., access to volume types, privileged): enable

fine-grained authorization of pod creation and update

◦ Network Policy (ingress, egress): control which pods can talk to each other

◦ Make Custom Resource Definitions namespace-scoped

◦ Sandboxes, e.g., gVisor

▫ Ensure security isolation of pods

▫ Ensure that information is not leaked between untrusted tenants

ISOLATION

◦ Resource Quotas (with Limit Range defaults) for CPU, memory, object

counts

◦ Pod Priority, Quality of Service Classes

◦ Node

▫ Taints & Tolerations: allow a node to repel a set of pods

▫ Labels and node selectors (less flexible than Affinity)

◦ Pod

▫ Affinity

▫ Pod Anti-affinity

FAIR SHARING - DATA PLANE (NODES)

DATA PLANE VS. CONTROL PLANE MULTITENANCY

Much of what we’ve talked about is sharing the data plane (nodes)

CONTROL PLANE MULTITENANCY

...Namespace 2

Namespace 1

master

User 2

CLI/API/UIUser 3

User 1

User n

Namespace n

Not able to manage multiple clusters →

1:1 mapping between master and cluster, though cluster may have multiple

tenants

MULTITENANT API SERVER?

...Namespace 2

Namespace 1

master

User 2

CLI/API/UIUser 3

User 1

User n

Namespace n

All tenants share master (incl. Secrets, ConfigMap), but RBAC helps

Little protection against individual tenants DoSing each other

MULTITENANT API SERVER?

Max inflight requests: mechanism for protecting API server against CPU and

memory overloads

Current problem to address: Tenants can crowd each other out (accidentally or

on purpose)

BETTER API SERVER FAIRNESS (UNDERWAY)

Proposal generalizes … max-in-flight request handler in the apiserver to make

more distinctions among requests and provide prioritization and fairness among

the categories of requests.”

BETTER API SERVER FAIRNESS (UNDERWAY)

Multiple priority levels, each has queues

Within each priority level, queues compete evenly

WHAT COMPANIES CARE ABOUT

Velocity

◦ Use multitenancy for improved resource efficiency, cost, and operations

◦ Different models of multitenancy

▫ Soft

▫ Hard

◦ Hard multitenancy is still work in progress

◦ Soft multitenancy is already used in production by various companies

▫ But is often coupled with a shared internal platform to gain

consistency of networking, security, etc. across teams

KEY TAKE-AWAYS

QUESTIONS?

◦ Project plan for multitenancy

▫ Building CRD for more automatic management of namespaces

◦ API Machinery KEP for improved resource sharing

◦ Cluster multi-tenancy for a good overview

Links for additional details

IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost,...

Documents

Transcript of IN KUBERNETES MULTITENANCY - qconsp.com · Use multitenancy for improved resource efficiency, cost,...

What s New in IBM Cognos Multitenancy(BP)

Topic Title API-first - qconsp.com fileEvery Business is Digital API enables Digital Sensedia powers your API Strategy IT Architecture Enabling Digital Transformation

MultiTenancy - an Introduction for Techies Timothy D. Kuehn Senior OpenEdge Consultant TDK Consulting Services Inc timk@tdkcs.catimk@tdkcs.ca tim.kuehn@gmail.com.

Oracle WebLogic 12c New Multitenancy features

for Distributed Systems - qconsp.com · Holden: My name is Holden Karau Prefered pronouns are she/her Developer Advocate at Google Apache Spark PMC, Beam contributor previously IBM,

Microservice Resiliency - qconsp.com › sp2017 › system › files › presentation-slides › lan… · Microservice Resiliency From Front to Back End QCon São Paulo, 2017 Lance

Multitenancy Options on Azure

CLOUD NATIVE@BMW GROUP...• Continuous Integration: Source code based deployment, automatic builds/deployments, staging • Tool completeness: user management, multitenancy support,

Understanding Multitenancy and the Architecture of the Salesforce Platform

Multitenant JVMwiki.jvmlangsummit.com/images/8/80/...Multitenancy... · Multitenancy is contrasted with a multi-instance architecture where separate software instances (or hardware

EDB WHITE PAPER Multitenancy Options in Postgres · multitenancy requirements for their application or system. When deciding which approach is most appropriate for a given environment,

Navision Stat 9.0 Multitenancy · Navision Stat 9.0 – Multitenancy CSY/ØSY/ RASEG 18.12.2017 Samlet opgraderings- og installationsvejledning Nedenstående beskriver den samlede

Introduction Scalability & Performance Security & Multitenancy Flexible Infrastructure High Availability & Resiliency Virtualization Innovation Summary.

AMIS Beyond the Horizon - High density deployments using weblogic multitenancy

High density deployments using weblogic multitenancy

VMWARE Professionals - Security, Multitenancy and Flexibility

Force.Com Multitenancy

OpenKM multitenancy

Dell Networking: Multitenancy with VRF-lite4 Dell Networking: Multitenancy with VRF-lite | Version 2.0 1 Overview The objective of this paper is to demonstrate multitenancy via VRF-lite

Think like an Elm developer - qconsp.com · About me The promise of Elm Background ... All languages must check types ... and code that runs error-free