NET1522BU
#VMworld #NET1522BU
Kubernetes Networking with NSX-T Deep Dive
VMworld 2017 Content: Not fo
r publication or distri
bution
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not been determined.
Disclaimer
CONFIDENTIAL 2
VMworld 2017 Content: Not fo
r publication or distri
bution
Yves Fauser / Yasen Simeonov
NET1522BU
#VMworld #NET1522BU
Kubernetes Networking with NSX-T Deep Dive
VMworld 2017 Content: Not fo
r publication or distri
bution
Agenda
1 NSX-T Overview
2 Kubernetes Overview
3 NSX-T & Kubernetes Integration
4 Demo
4#NET1522BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX Vision: Driving NSX Everywhere
#NET1522BU CONFIDENTIAL 6
Managing security and connectivity for many heterogeneous end points
New app frameworks
Branch offices/Edge computing/IOT
End usersOn-premises data center
Automation
IT at the speed of business
Security
Inherently secure infrastructure
Application Continuity
Data center anywhere
Cloud
vCloud AirNetwork
VMworld 2017 Content: Not fo
r publication or distri
bution
Transport Nodes
NSX Manager
NSX Controllers
NSX-T Architecture
#NET1522BU CONFIDENTIAL 7
NSX Architecture and Components
Cloud Consumption• Self Service Portal
• OpenStack, K8s, Custom
Data Plane
• High Performance Data Plane
• Scale-out Distributed Forwarding Model
Management Plane (MP) Node – VM form factor
• Concurrent configuration portal
• REST API entry-point
• UI
Central Control Plane (CCP) Nodes- VM form factor
• Talks to Dataplane over a Control-Plane
Protocol
• Separation of Control and Data Plane
ESXi (+ kernel modules)
Control Plane
Management Plane
NSX Edge(L3 + Adv
Services)
Physical Infrastructure
Hypervisors
L2 Bridge(L2 Overlay-
VLAN)
KVM(+ kernel modules)
VMworld 2017 Content: Not fo
r publication or distri
bution
CCP Node CCP Node CCP Node
MP Node
NSX-T Architecture
#NET1522BU CONFIDENTIAL 8
Operations Workflow
User makes a configuration
Transport
Node
MPA LCP
Transport
Node
MPA LCP
Transport
Node
MPA LCP
XConfiguration is “persisted”
Configuration is pushed to CCP
Configuration is realized
VMworld 2017 Content: Not fo
r publication or distri
bution
Data Plane
#NET1522BU CONFIDENTIAL 9
Improved performance and resiliency
Admin
Tenants/CMP
Designed for multi-tenancy and scale
New distributed edge architecture with increased performance with
DPDK
p1 p2
HV TN1vSwitch
1
TEP
Overlay Transport Zone
TEP: Overlay Tunnel End Point
(with its own IP address)
GENEVE Tunnel
p1 p2
HV TN1vSwitch
2
TEP
Next gen overlay maintaining performance with increased flexibility
Edge
Node
Edge Cluster
Edge
Node
Edge
Node
Edge
Node
VMworld 2017 Content: Not fo
r publication or distri
bution
#NET1522BU CONFIDENTIAL 10
NSX-T Breakout Session
VMware NSX-T - Getting Started
SPL182601U (US)
SPL182601E (Europe)
NSX-T Hands On Lab
Introduction to NSX-T Architecture NET1510BU (US)
NET1510BE (Europe)
NSX-T VMworld Session & Lab
VMworld 2017 Content: Not fo
r publication or distri
bution
#NET1522BU CONFIDENTIAL 12
Kubernetes is an open-source platform for automating deployment, scaling, and operations of
application containers across clusters of hosts, providing container-centric infrastructure.
What Is Kubernetes?
VMworld 2017 Content: Not fo
r publication or distri
bution
• K8s Cluster Consists of Master(s) and Nodes
• K8s Master Components
– API Server
– Scheduler
– Controller Manager
– Dashbord
• K8s Node Components
– Kubelet
– Kube-Proxy
– Containers Runtime (Docker or Rocket)
#NET1522BU CONFIDENTIAL 13
K8s masterK8s master
K8s
Master
Controller
Manager
K8s API
Server
Key-Value
Store
dashboard
Scheduler
K8s nodeK8s node
K8s nodeK8s node
K8s Nodes
kubelet c runtime
Kube-proxy
> _ Kubectl
CLI
K8s Master(s)
Kubernetes Components
VMworld 2017 Content: Not fo
r publication or distri
bution
#NET1522BU CONFIDENTIAL 14
Pod
pause container(‘owns’ the IP stack)
10.24.0.0/16
10.24.0.2
nginxtcp/80
mgmttcp/22
loggingudp/514
IPC
External IP Traffic
A Pod is a group of one or more containers that shares an IP address and a Data Volume
Kubernetes Pod
VMworld 2017 Content: Not fo
r publication or distri
bution
#NET1522BU CONFIDENTIAL 15
Namespace: fooBase URI: /api/v1/namespaces/foo
‚redis-master‘ Pod:/api/v1/namespaces/foo/pods/redis-master
‚redis‘ service:/api/v1/namespaces/foo/services/redis-master
Namespace: barBase URI: /api/v1/namespaces/bar
‚redis-master‘ Pod:/api/v1/namespaces/bar/pods/redis-master
‚redis‘ service:/api/v1/namespaces/bar/services/redis-master
• Namespaces are a way to divide cluster resources between multiple uses
• They can be considered as Tenants
• They are a way to provide Resources Quotas, RBAC, Networking Multitenancy, and Names Overlapping
Kubernetes Namespace
VMworld 2017 Content: Not fo
r publication or distri
bution
East-West Load Balancing North-South Load Balancing
#NET1522BU CONFIDENTIAL 16
Redis Slave
Pods
redis-slave svc
10.24.0.5/16
172.30.0.24
Web Front-End
Pods
East-West Load Balancing is provided through K8s Service using ClusterIP & IPTables
Web Front-End
(e.g. Apache) Pods
Web Front-End
IngressNginx || HAProxy || etc.
LB Pods
http://*.bikeshop.com
Can be achieved through K8s Ingress or External third Party Load Balancer using NodePort
K8s Load Balancing
VMworld 2017 Content: Not fo
r publication or distri
bution
Nodeint eth0
10.240.0.4
int cbr0
10.24.2.1/24
10.24.2.2 10.24.2.3 10.24.2.4
Kubernetes Networking Topologies
#NET1522BU CONFIDENTIAL 17
Flat routed topology
Nodeint eth0
10.240.0.3
int cbr0
10.24.1.1/24
10.24.1.2 10.24.1.3 10.24.1.4
net.ipv4.ip_forward=1
net.ipv4.ip_forward=1
• Every Node is an IP Router and responsible for its Pod Subnet
• Subnets are associated with Nodes, not Tenants
• Physical Network Configuration is required
ip route 10.24.1.0/24 10.240.0.3
ip route 10.24.2.0/24 10.240.0.4
VMworld 2017 Content: Not fo
r publication or distri
bution
Kubernetes Networking Topologies
#NET1522BU CONFIDENTIAL 18
Node-to-Node overlay topology
Nodeint eth0
10.240.0.4
int cbr0
10.24.2.1/24
10.24.2.2 10.24.2.3 10.24.2.4
Nodeint eth0
10.240.0.3
int cbr0
10.24.1.1/24
10.24.1.2 10.24.1.3 10.24.1.4
net.ipv4.ip_forward=1
net.ipv4.ip_forward=1
Overlay
Key-Value
Store
Overlays are typically used to avoid Physical Network Configuration
VMworld 2017 Content: Not fo
r publication or distri
bution
#NET1522BU CONFIDENTIAL 20
admin@k8s-master:~$ kubectl create namespace foonamespace ”foo" created
admin@k8s-master:~$ kubectl create namespace barnamespace ”bar" created
admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foodeployment "nginx-foo" created
admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bardeployment "nginx-bar" created
Namespace: foo Namespace: bar
NSX / K8s topology
10.24.0.0/24 10.24.1.0/24 10.24.2.0/24
NAT boundary
NAT boundary
K8s nodesK8s Masters
NSX-T K8s Integration – Namespaces & Pods
VMworld 2017 Content: Not fo
r publication or distri
bution
#NET1522BU CONFIDENTIAL 21
admin@k8s-master:~$ vim no-nat-namespace.yaml
apiVersion: v1kind: Namespacemetadata:
name: no-nat-namespaceannotations:
ncp/no_snat: "true“
admin@k8s-master:~$ kubectl create –f no-nat-namespace.yamlnamespace ”no-nat-namespace" created
admin@k8s-master:~$ kubectl run nginx-no-nat --image=nginx –n no-nat-namespacedeployment "nginx-k8s" created
Namespace: no-nat-namespace
NSX / K8s topology
114.4.10.0/26
Direct Routing
114.4.10.64/26
K8s nodesK8s Masters
NSX-T K8s Integration – Routed Namespaces
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX-T K8s Integration – Pods Micro-Segmentations
#NET1522BU CONFIDENTIAL 22
Option1: Predefined Label Based Rules
admin@k8s-master:~$ kubectl label pods nginx-foo-3492604561-nltrf secgroup=web -n fooPod "nginx-nsx-3492604561-nltrf" labeled
admin@k8s-master:~$ kubectl label pods nginx-bar-2789337611-z09x2 secgroup=db -n barpod "nginx-k8s-2789337611-z09x2" labeled
admin@k8s-master:~$ kubectl get pods --all-namespaces -LsecgroupNAMESPACE NAME READY STATUS RESTARTS AGE SECGROUPk8s nginx-foo-2789337611-z09x2 1/1 Running 0 58m webnsx nginx-bar-3492604561-nltrf 1/1 Running 0 1h db
Namespace: foo Namespace: bar
NSX / K8s topology
10.24.0.0/24 10.24.1.0/24 114.4.10.0/26
NAT boundary
NAT boundary
Web
• Security Groups are defined in NSX with ingress and egress policy
• Each Security Group could be micro-segmented to protect Pods from each other
DB
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX-T K8s Integration – Pods Micro-Segmentations
#NET1522BU CONFIDENTIAL 23
Option 2: K8s Network Policy
admin@k8s-master:~$ vim nsx-demo-policy.yamlapiVersion: extensions/v1beta1kind: NetworkPolicymetadata:name: nsx-demo-policy
spec:podSelector:matchLabels:app: web
ingress:- from:
- namespaceSelector:matchLabels:ncp/project: db
ports:- port: 80protocol: TCP
admin@k8s-master:~$ kubectl create -f nsx-demo-policy.yaml
• State: released on K8s 1.7 (Beta on 1.6)
• Capability: Using Network Policy, users can define firewall rules to allow traffic into and out of a Namespace, and between Pods. The network policy is a Namespace property. The default is drop.
Namespace: foo Namespace: bar
NSX / K8s topology
10.24.0.0/24 10.24.1.0/24 114.4.10.0/26
NAT boundary
Routed
DB
Label: app=db
Web
Label: app=web
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX-T K8s Integration – Pods Micro-Segmentations
#NET1522BU CONFIDENTIAL 24
Option 2: K8s Network Policy
$ kubectl create -f nsx-demo-policy.yaml
Dynamic Creation of Security Groups
Dynamic Creation of Security Policy based on k8s Network Policy
Once the Network Policy is applied, NSX will dynamically create source & destination Security Groups and apply the right policy
VMworld 2017 Content: Not fo
r publication or distri
bution
• Micro-Segmentation in K8s: The data model to describe segmentation policies between Namespaces, and within namespaces is called ’Network Policies’ and is released on Kubernetes 1.7 (Beta on 1.6)
Firewalling in Kubernetes
• NSX could utilize K8s Network Policies to define Dynamic Security Groups & Policies
• Capabilities are limited to K8s Network Policy capabilities
K8s Network Policy
• Security Groups & Policies could be predefined on NSX. Labels are used to specify Pods Membership
• Mapping of IP based groups, egress rules, VM based matching could be available to be used in the policy definition
Pre-Defined Label based rules
• The NSX / K8s integration intends to support both the pre-defined label based rules and K8s network policy
Firewalling in NSX / K8s
#NET1522BU CONFIDENTIAL 25
NSX-T K8s Integration – Pods Micro-Segmentations
VMworld 2017 Content: Not fo
r publication or distri
bution
#NET1522BU CONFIDENTIAL 26
Node
VM
NSX CNI
Plugin
OVS
Pods
NSX KubeProxy
K8s masterK8s master
K8s
Master
Controller
Manager
K8s API
Server
dashboard
Scheduler
• K8s Services are delivered through NSX Kube-Proxy
• Delivered as a container image, so that it can be run as a Kubernetes Daemon-Set on the Nodes
• NSX Kube-Proxy would replace the native distributed east-west load balancer in Kubernetes called Kube-Proxy
• OpenVSwitch (OVS) load-balancingis used
East-West Load Balancing
VMworld 2017 Content: Not fo
r publication or distri
bution
Once an Ingress Controller is added, NSX will define SNAT & DNAT rules
#NET1522BU CONFIDENTIAL 27
Web Front-End
Ingress
Nginx
Ingress LB Pod
http://*.demo.corp.local
10.4.0.0/24 10.4.1.0/24
10.4.0.67
North-South Load Balancing
VMworld 2017 Content: Not fo
r publication or distri
bution
• NCP is a software component provided by VMware in form of a container image, e.g. to be run as a K8s Pod
• NCP is build in a modular way, so that individual adapters can be added for different CaaS and PaaS systems
K8s / NSX Components
#NET1522BU CONFIDENTIAL 28
NSX Container Plugin (NCP)
VMworld 2017 Content: Not fo
r publication or distri
bution
Namespace creation workflow
NCP
Infra
K8s
Adapter
NSX Container Plugin
NSX
Manager
API Client
NSX
Manager
NS: foo
NSX/ K8s topology
NS: bar
K8s master
etcd
API-
Server
Scheduler
1)2)
3)
4)
1. NCP creates a ‚watch‘ on K8s API for any Namespace events
2. A user creates a new K8s Namespace
3. The K8s API Server notifies NCP of the change (addition) of Namespaces
4. NCP creates the network topology for the Namespace:
a) Requests a new subnet from the pre-configured IP block in NSX
b) Creates a logical switchc) Creates a T1 router and attaches it to
the pre-configured global T0 routerd) Creates a router port on the T1 router,
attaches it to the LS, and assigns an IP from the new subnet
K8s / NSX Workflows
#NET1522BU CONFIDENTIAL 29
Namespace / Topology creation
VMworld 2017 Content: Not fo
r publication or distri
bution
Hypervisor
(ESXi &
KVM)
Node
VM
DFW
eth2
Node
VM
DFW
eth0
Minion Mgmt.
IP Stack
eth0
Minion Mgmt.
IP Stack
mgmtnetwork
OVS
mgmtnetwork
Vla
n10
vla
n11
cifcif
eth2
vla
n10
vla
n11
OVS
cifcif
NSX CNI
Plugin
NSX CNI
Plugin
Pods
Pods
#NET1522BU CONFIDENTIAL 30
• Management Interface is Separated
from the interface used for Pods traffic
• CIF is used per K8s Pod
• CIFs are differentiated through locally
significant VLAN tags
• NSX CNI Plugin is responsible for
tagging the traffic with the right VLAN
• NCP will map the VLAN tags to a
specific CIF
NSX-T Container Interface (CIF)
VMworld 2017 Content: Not fo
r publication or distri
bution
#NET1522BU CONFIDENTIAL 31NSX-T Traceflow
NSX-T Operational Tools• Traceflow• Port Mirroring• Port Connection Tool• Spoofguard• Syslog• Port Counters• IPFIX
NSX-T Operational Tools for K8s
VMworld 2017 Content: Not fo
r publication or distri
bution
#NET1522BU CONFIDENTIAL 33
Enterprise-class
Networking
Advanced Security Enhanced
Operations
Full Network
Visibility
Enterprise
Support
Unified VM-to-
Pod
Networking
Pods Micro-
Segmentation
N S X - T Va l u e s f o r K 8 s
F e a t u r e s
NSX-T Values for K8s
VMworld 2017 Content: Not fo
r publication or distri
bution
Hands On Lab Self-Paced Lab
VMware NSX-T with KubernetesSPL182602U(US)
SPL182602E(Europe)
Kubernetes and VMware NSX Blog
https://blogs.vmware.com/networkvirtualization/2017/03/
kubecon-2017.html/#NET1522BU CONFIDENTIAL 34
VMworld 2017 Content: Not fo
r publication or distri
bution
Join VMUG for exclusive access to NSX
vmug.com/VMUG-Join/VMUG-Advantage
Connect with your peers
communities.vmware.com
Find NSX Resources
vmware.com/products/nsx
Network Virtualization Blog
blogs.vmware.com/networkvirtualization
Where to Get Started
Dozens of Unique NSX Sessions
Spotlights, breakouts, quick talks & group discussions
Visit the VMware Booth
Product overview, use-case demos
Visit Technical Partner Booths
Integration demos – Infrastructure, security, operations,
visibility, and more
Meet the Experts
Join our Experts in an intimate roundtable discussion
Free Hands-on Labs
Test drive NSX yourself with expert-led or self-paces
hands-on labs
labs.hol.vmware.com
Training and Certification
Several paths to professional certifications. Learn
more at the Education & Certification Lounge.
vmware.com/go/nsxtraining
Engage and Learn Experience
Try Take
#NET1522BU CONFIDENTIAL 35
VMworld 2017 Content: Not fo
r publication or distri
bution
Top Related