#vmworld
Deep Dive: VMware Cloud PKS
K8s as a service on Public Cloud
Valentina AlariaDirector of Product Management, VMware, Inc.
Tom SpoonemoreProduct Manager, VMware, Inc.
CNA3124BE
#CNA3124BE
VMworld 2018 Content: Not for publication or distribution
Disclaimer
2©2018 VMware, Inc.
This presentation may contain product features orfunctionality that are currently under development.
This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.
VMworld 2018 Content: Not for publication or distribution
3©2018 VMware, Inc.
VMware Kubernetes Engine is now …
VMworld 2018 Content: Not for publication or distribution
Agenda
4©2018 VMware, Inc.
VMware Cloud PKS: Intro & Architecture
VMware Smart Cluster: Intro & Deep Dive
Connections
Policy Framework
Demo
Q&A
VMworld 2018 Content: Not for publication or distribution
5©2018 VMware, Inc.
Kubernetes makes it easy to manage applications that are container-based.
Easy to deploy your application• Applications are composed of pods that are sets of containers running on the same host• Applications are easily spread across multiple nodes
Easy to scale your application• You can easily increase/decrease the number of instances of your application• You can automate the scaling of your application easily
Easy-to-use network connectivity• Kubernetes makes it easy to deploy load balancers in front of your application• Network policies allow protection between applications deployed in a cluster
Easy-to-use storage• Persistent volumes can be dynamically allocated and used by your application
A quick reminder: What is Kubernetes?
VMworld 2018 Content: Not for publication or distribution
6©2018 VMware, Inc.
Kubernetes Topology:• A set of master nodes manage the cluster• A set of worker nodes run your application
Kubernetes applications:• Pod: a container (or set of containers) run on one
nodeThink: One instance of your application
• ReplicaSet: A set of identical pods:Think: Your application
• Service: Acts like a load balancer for a set of podsCan be internal (between pods)Can be external (allow access from clients)
Kubernetes
Master Master Master
Worker Worker Worker
Service: Your app
Pod Pod Pod
VMworld 2018 Content: Not for publication or distribution
7©2018 VMware, Inc.
VMware’s Kubernetes PortfolioMeeting customers wherever they run their apps on any infrastructure
Enterprise Software Cloud Service
PublicBeta
VMware Cloud PKS VMware PKS
VMworld 2018 Content: Not for publication or distribution
8©2018 VMware, Inc.
US West
US East
Europe West
An enterprise-grade Kubernetes-as-a-Service offering in the VMware Cloud Services portfolio.
• Pay by the second, on-demand over public internet
• Launching on multiple AWS regions, with support for Azure and other platforms in the future
• Globally consistent policy management
• Certified Kubernetes conformant• Full integration with AWS services
VMware Cloud PKSHighly Secure and Available Kubernetes Service
VMworld 2018 Content: Not for publication or distribution
9©2018 VMware, Inc.
VMware Cloud PKS Part of VMware Cloud Services, Offered as a SaaS-based Model
9
VMware Cloud Services (cloud.vmware.com)
• Single Sign-on
• Single Bill to Manage
• Single Global Support
VMware Cloud PKS
Enterprise grade Kubernetes-as-a-Service offering that provides easy to use, secure by default, and highly efficient containers.
VMworld 2018 Content: Not for publication or distribution
VMware Cloud PKS: Production-Grade K8s for Enterprise
Easy to Use and Maintain
Highly Secure by Default
Integrates with EcosystemMulti-tenancy
• Multi-AZ Master and Etcd
• 7x24 Continuous Health Monitoring of entire cluster
• Auto-remediation of all issues
Highly Available “Dial Tone” Kubernetes
• Hardened Kubernetes
• Continuously patched
• Data encrypted in transit and at rest
• Role and resourced based access policies
• Each customer in isolated AWS, Azure account
• Powerful policy framework across all cloud platforms and all regions
• 90 seconds from account activation to first cluster
• No training or staffing required
• Fully tested and documented integrations with Jenkins, Prometheus, Istio and others
• Integration with AWS services over private networkVMworld 2018 Content: Not for publication or distribution
11©2018 VMware, Inc.
Kubernetes Ecosystem
Native Kubernetes compatibility with leading open-source solutions and tools
Commercial Partners & Solutions
Broadening feature portfolio & solutions for application deployment
VMware Cloud Services
Simple and easy integration with other VMware Cloud Services
VMware Cloud PKS: Integration EcosystemBroad Ecosystem of Application Building Blocks
WavefrontCode Stream
VMworld 2018 Content: Not for publication or distribution
12©2018 VMware, Inc.
• Service Meshes: microservice network layer for handling service-to-service communication
• Declarative policies for application observability, traffic management, and security functionalities
• Istio brings enhanced set of L7 metrics used for application monitoring, logging and tracing by Operators & Users
• Istio powers advanced traffic routing and management• Canary Deployments• Rate Limiting• Circuit Breakers• Authentication
Announcing Service Mesh & ObservabilityVMware Cloud PKS Validated Solution: Istio
VMworld 2018 Content: Not for publication or distribution
13©2018 VMware, Inc.
VMware Cloud PKS Architecture
VMworld 2018 Content: Not for publication or distribution
14©2018 VMware, Inc.
VMware Cloud PKS Architecture
AWS: us-west-2
Clusters
AWS: eu-west-1
Clusters
Azure: West US
Clusters
VMware Cloud PKS Management Plane
Fully distributed, highly available management plane• Each region shares config data but operates independently• Deployed across multiple availability zonesIsolated, highly available clusters• Each cluster within a single region, spread across availability zones• Each cluster isolated within private network• Each customer isolated within distinct cloud provider account, managed by VMware Cloud PKS
…
VMworld 2018 Content: Not for publication or distribution
15©2018 VMware, Inc.
Smart Cluster Introduction
VMworld 2018 Content: Not for publication or distribution
16©2018 VMware, Inc.
VMware Smart Cluster
VMware Smart Cluster automates selection of compute resources constantly optimizing resource usage and reducing cost.
• Removes need for educated guesses around cluster definition and sizing
• Enables management of cost-effective, scalable Kubernetes clusters that are constantly optimized to application needs.
• Provides built-in resiliency with routine health checks and self-healing capabilities for Kubernetes clusters.
• Makes it seamless for a user to run and/or manage highly availabledeployments without additional cost and complexity.
VMware Smart Cluster Run Kubernetes without Managing Servers or Clusters
VMworld 2018 Content: Not for publication or distribution
18©2018 VMware, Inc.
Smart Cluster Deep Dive
VMworld 2018 Content: Not for publication or distribution
19©2018 VMware, Inc.
Production ClusterDetail View
Namespaces vke-system kube-system default
Pods(vke-system)
Smart cluster monitor Ingress CNI
Availability Zone A
Master Node
+etcd
Worker Nodes
Availability Zone B
Availability Zone C
Master Node
+etcd
Worker Nodes
Master Node
+etcd
Worker Nodes
Namespaces vke-system kube-system default
Pods(vke-system)
Smart cluster monitor Ingress CNI
Availability Zone A
Master Node
+etcd
Worker Nodes
Availability Zone B
Availability Zone C
Master Node
+etcd
Worker Nodes
Master Node
+etcd
Worker Nodes
Production Cluster 1
• Clusters in managed, isolated AWS account
• Each cluster is in a single region
• Each cluster is in an isolated VPC
• Clusters are fully HA:
• Multiple masters
• Three availability zones
Production Cluster 2
VMworld 2018 Content: Not for publication or distribution
20©2018 VMware, Inc.
Development ClusterDetail View
• Clusters in managed, isolated AWS account
• Each cluster in a single region
• Shared VPC for multiple Development Clusters
• Developer sandbox or test environments
• Single Master node per cluster
Namespaces vke-system
kube-system default
Pods(vke-
system)
Smart cluster monitor Ingress CNI
Master Node
+etcd
Worker Nodes
Worker Nodes
Worker Nodes
Availability Zone
Namespaces vke-system
kube-system default
Pods(vke-
system)
Smart cluster monitor Ingress CNI
Master Node
+etcd
Worker Nodes
Worker Nodes
Worker Nodes
Dev Cluster 1 Dev Cluster 2
VMworld 2018 Content: Not for publication or distribution
21©2018 VMware, Inc.
• Smart cluster monitor watches pods and nodes and contacts VMware Cloud PKS management plane when changes are needed.
• When a pod is created and cannot be scheduled, VMware Cloud PKS creates a new worker node.
• When pods are deleted and nodes are underutilized, VMware Cloud PKS deletes worker nodes and pods are moved as necessary
ElasticityElastic Smart Clusters
Smart Cluster
Master Worker Worker
Developers
VMware Cloud PKS
Smart Cluster Monitor
Kubernetes Scheduler
Kubernetes Pod Pending
Worker
Deployments
Kubernetes API
Pods
Clo
ud A
PI
VMworld 2018 Content: Not for publication or distribution
22©2018 VMware, Inc.
• Managed OS patching for Kubernetes nodes
• On demand upgrades for Kubernetes clusters
OS Patches and Kubernetes Upgrades
Automated PatchingKubernetes Nodes OS Patching
What• Patches will be applied within 7
days from the time the patch is available for all Critical Vulnerability CVE’s >9
• 14 days for CVE’s >7.5 and <9
How• Patches applied automatically by
VMware Cloud PKS• Nodes rebooted one at a time• Production Smart Clusters have zero-
downtime because there are three masters.
Kubernetes Upgrade What
• Patch Releases: 2 weeks after release• Minor Releases : 4 weeks after release• Major Releases: TBD
How• Kubernetes upgrades initiated by customers• Rolling updates applied to cluster• Master nodes get upgraded first• etcd’s are backed up before upgrading• Cluster is rolled back if upgrade fails
VMworld 2018 Content: Not for publication or distribution
23©2018 VMware, Inc.
- Data Encryption at rest and in Motion
- Encrypted Kubernetes secrets in etcd
Encryption
Data Encryption
VM
war
e C
loud
PK
S
User profile
Application data
Policy dataUsers
Applications
Https
Https
- TLS Encryption for traffic- From Users to VMware Cloud PKS- Between masters, worker and etcVMworld 2018 Content: Not for publication or distribution
24©2018 VMware, Inc.
Health Monitoring
Each Smart Cluster actively monitored• Capacity• Faults
etcd Kubernetes API Server
Kubernetes Scheduler
Kubernetes UI
# of Master and Worker
nodes
Smart Cluster Monitor
Smart Cluster Scaler
K8s Dashboard KubeDNS CNI
Manager
VMware Cloud PKS Service Pods
Kubernetes Cluster
Health Metrics Stream
Metrics
VMware Cloud PKS Ops Team Remediation
BotsVMworld 2018 Content: Not for publication or distribution
25©2018 VMware, Inc.
Smart Cluster Summary
Features Development Smart Cluster
Production Smart Cluster
Pod Networking X X
Elasticity X X
Managed OS Upgrades X X
Health Monitoring X X
Validated Solutions X X
Zero-Downtime Upgrade X
VPC Network Isolation X
High Availability X
Connections X
VMworld 2018 Content: Not for publication or distribution
27©2018 VMware, Inc.
AWS Service Access using VPC Peering
Customer Deployed Virtual Private Cloud (VPC) Assets
EC2 Instance Applications
Private Load Balancer Listeners
Proxied Services
AWS PrivateLink Endpoint Services
Peer VPC Private DNS (in-region peers only)
Amazon EC2
Amazon VPC
AmazonS3
Amazon EFS
AmazonRDS
AmazonDynamoDB
AmazonRoute 53
AWS Service Ecosystem
And many others …
VMworld 2018 Content: Not for publication or distribution
28©2018 VMware, Inc.
Customer AWS
Account
Cloud PKS AWS
Shadow Account
Connecting your existing VPCSimple Access to AWS Services
Subnet :10.16.1.0/24VPC
AWS A/C: cpks-shadow-tenant-A
VPC peering
Subnet:.10.0.0.0/24
VPC
AmazonRDS
AWS A/C: Peer User Account
Your Existing AWS Account and VPC• Self-managed Deployment• Integrated AWS Services
Your Cloud PKS Account• Cloud PKS managed VPC• Production Smart Cluster
Connections initiated VPC Peer
VMworld 2018 Content: Not for publication or distribution
29©2018 VMware, Inc.
Features
• L3 Connectivity to other AWS Account VPCs
• Supports cluster egress traffic today
• Traffic routed internally on AWS private networks
• Security Groups protects cluster from unauthorized access
• Private DNS name resolution supported within region
• Traffic charged at inter-AZ or inter-region rates
Considerations
• VPC IP CIDR blocks cannot overlap
• Max of 50 active peering connections / VPC
What to know about VPC Peering?
Customer AWS
Account
Cloud PKS AWS
Shadow Account
Subnet :10.16.1.0/24VPC
AWS A/C: cpks-shadow-tenant-A
VPC peering
Subnet:.10.0.0.0/24
VPC
AmazonRDS
AWS A/C: Peer User Account
VMworld 2018 Content: Not for publication or distribution
32©2018 VMware, Inc.
Access Policy Framework
CONFIDENTIAL
Organization
…
Folder 1 Folder 2
Project 1 Project 2 Project 3
Cluster 1 Cluster 2
Namespace 1 Namespace 2
Cluster X
Namespace Y
Each customer is mapped to an organization.
An organization is hierarchical.
Access policies are applied anywhere, and inherited down the hierarchy.
Access policies are pushed to Kubernetes cluster and enforced within Kubernetes
VMware Cloud PKS Logical Resources
Kubernetes ResourcesVMworld 2018 Content: Not for publication or distribution
33©2018 VMware, Inc.
Access Policy FrameworkAccess Policy & Defaults
• Access Policies can be defined at any node of the hierarchy tree
• All Users by default have access to “Shared Folder” and “Shared Project” in each Organization
• By default the Organization has below policy
Node Role User/Group
Org *.iam.edit, *.edit, smartcluster.admin
VKEServiceAdministrators
Org Org.view, Group.view
VKEServiceUsers
Shared Folder
Folder.view VKEServiceUsers
Shared Project
Project.edit VKEServiceUsers
Shared Project
SmartCluster.admin VKEServiceUsers
Organization
…
Folder 1 Shared Folder
Project 1 Project 2 Shared Project
Cluster 1 Cluster 2
Namespace 1 Namespace 2
Cluster X
Namespace Y
VMworld 2018 Content: Not for publication or distribution
34©2018 VMware, Inc.
Granting Alice the “SmartCluster.Edit” role on a folder gives her SmartCluster.Edit on all clusters within that folder.
Access Policy Example #1Enabling a user to access all clusters in a folder
Organization
…
Folder 1
Edit:Alice
Folder 2
Project 1 Project 2 Project 3
Cluster 1
Edit:Alice
Cluster 2
Edit:Alice
Namespace 1 Namespace 2
Cluster X
Namespace Y
Alice is granted accessto all clusters in folder 1
Customer’s Logical View
VMworld 2018 Content: Not for publication or distribution
35©2018 VMware, Inc.
Granting Bob access in a project gives him access to all clusters in that project.
Access Policy Example #2: ProjectEnabling a user to access all clusters in a project
Organization
…
Folder 1 Folder 2
Project 1 Project 2 Project 3
Cluster 1 Cluster 2
Namespace 1 Namespace 2
Cluster X
Namespace Y
Customer’s Logical View
Bobis granted accessto all clusters in Project 1
Edit:Bob Edit:Bob
VMworld 2018 Content: Not for publication or distribution
36©2018 VMware, Inc.
Granting Carol access to a single cluster gives her access to only that cluster.
Access Policy Example #3: ClusterEnabling a user to access just one cluster
Organization
…
Folder 1 Folder 2
Project 1 Project 2 Project 3
Cluster 1 Cluster 2
Namespace 1 Namespace 2
Cluster X
Namespace Y
Customer’s Logical View
Edit:Carol
Carol is granted accessonly to cluster 1
VMworld 2018 Content: Not for publication or distribution
37©2018 VMware, Inc.
Granting David access to a single namespace only grants access to that single namespace in a single cluster.
Access Policy Example #4: NamespaceEnabling a user to access just one namespace in one cluster
Organization
…
Folder 1 Folder 2
Project 1 Project 2 Project 3
Cluster 1 Cluster 2
Namespace 1 Namespace 2
Cluster X
Namespace Y
Customer’s Logical View
David is granted accessonly to one namespace
in Cluster 1
Edit:David
VMworld 2018 Content: Not for publication or distribution
39©2018 VMware, Inc.
Visit cloud.vmware.com Request Access Log onto console.cloud.vmware.com
and start using service
Getting Started with VMware Cloud Services is Easy
VMworld 2018 Content: Not for publication or distribution
©2018 VMware, Inc.
Thank you!Questions?Valentina Alaria: [email protected] Spoonemore: [email protected]
VMworld 2018 Content: Not for publication or distribution
DON’T FORGET TO FILL OUT YOUR SURVEY.
#vmworld #CNA3124BE
VMworld 2018 Content: Not for publication or distribution
Top Related