NET1522BE Kubernetes Networking with NSX-T … Al Idrees Yves Fauser NET1522BE #VMworld #NET1522BE...

Ali Al IdreesYves Fauser

NET1522BE

#VMworld #NET1522BE

Kubernetes Networking with NSX-T Deep Dive

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

2#NET1522BE CONFIDENTIAL



bution

Agenda

1 NSX-T Overview

2 Kubernetes Overview

3 NSX-T & Kubernetes Integration

4 Demo




bution

NSX-T Overview



bution

5

NSX Vision: Driving NSX everywhereManaging security and connectivity for many heterogeneous end points

New app frameworks

Branch offices/Edge computing/IOT

End users

On-premises data center

Automation

IT at the speed of business

Security

Inherently secure infrastructure

Application Continuity

Data center anywhere

Cloud

vCloud AirNetwork



bution

Transport Nodes

NSX Manager

NSX Controllers

NSX-T ArchitectureNSX Architecture and Components

Cloud Consumption• Self Service Portal

• OpenStack, K8s, Custom

Data Plane

• High Performance Data Plane

• Scale-out Distributed Forwarding Model

Management Plane (MP) Node – VM form factor

• Concurrent configuration portal

• REST API entry-point

• UI

Central Control Plane (CCP) Nodes- VM form factor

• Talks to Dataplane over a Control-Plane

Protocol

• Separation of Control and Data Plane

ESXi(+ kernel modules)

Control Plane

Management Plane

NSX Edge(L3 + Adv

Services)

Physical Infrastructure

Hypervisors

L2 Bridge(L2 Overlay-

VLAN)

KVM(+ kernel modules)




bution

CCP Node CCP Node CCP Node

MP Node

NSX-T ArchitectureOperations Workflow

7

User makes a configuration

Transport

Node

MPA LCP

Transport

Node

MPA LCP

Transport

Node

MPA LCP

XConfiguration is “persisted”

Configuration is pushed to CCP

Configuration is realized

#NET1522BE CONFIDENTIAL



bution

Data Plane

Improved performance and resiliency

Admin

Tenants/CMP

Designed for multi-tenancy and scale

New distributed edge architecture with increased performance with

DPDK

p1 p2

HV TN1vSwitc

h1

TEP

Overlay Transport Zone

TEP: Overlay Tunnel End Point

(with its own IP address)

GENEVE Tunnel

p1 p2

HV TN1vSwitc

h2

TEP

Next gen overlay maintaining performance with increased flexibility

Edge

Node

Edge Cluster

Edge

Node

Edge

Node

Edge

Node




bution

NSX-T VMworld Session & Lab

9

NSX-T Breakout Session

VMware NSX-T - Getting Started

SPL182601U (US)

SPL182601E (Europe)

NSX-T Hands On Lab

Introduction to NSX-T Architecture NET1510BU (US)

NET1510BE (Europe)




bution

Kubernetes Overview



bution

What is Kubernetes?

Kubernetes is an open-source platform for automating deployment, scaling, and operations of

application containers across clusters of hosts, providing container-centric infrastructure.




bution

Kubernetes Components

• K8s Cluster Consists of Master(s) and Nodes

• K8s Master Components

– API Server

– Scheduler

– Controller Manager

– Dashbord

• K8s Node Components

– Kubelet

– Kube-Proxy

– Containers Runtime (Docker or Rocket)

12

K8s masterK8s master

K8s

Master

Controller

Manager

K8s API

Server

Key-Value

Store

dashboard

Scheduler

K8s nodeK8s node

K8s nodeK8s node

K8s Nodes

kubelet c runtime

Kube-proxy

> _ Kubectl

CLI

K8s Master(s)




bution

Kubernetes Pod

Pod

pause container(‘owns’ the IP stack)

10.24.0.0/16

10.24.0.2

nginxtcp/80

mgmttcp/22

loggingudp/514

IPC

External IP Traffic

• A Pod is a group of one or more containers that shares an IP address and a Data Volume




bution

Kubernetes Namespace

Namespace: fooBase URI: /api/v1/namespaces/foo

‚redis-master‘ Pod:/api/v1/namespaces/foo/pods/redis-master

‚redis‘ service:/api/v1/namespaces/foo/services/redis-master

Namespace: barBase URI: /api/v1/namespaces/bar

‚redis-master‘ Pod:/api/v1/namespaces/bar/pods/redis-master

‚redis‘ service:/api/v1/namespaces/bar/services/redis-master

• Namespaces are a way to divide cluster resources between multiple uses

• They can be considered as Tenants

• They are a way to provide Resources Quotas, RBAC, Networking Multitenancy, and Names Overlapping




bution

K8s Load Balancing

East-West Load Balancing North-South Load Balancing

15

Redis Slave

Pods

redis-slave svc

10.24.0.5/16

172.30.0.24

Web Front-End

Pods

East-West Load Balancing is provided through K8s Service using ClusterIP & IPTables

Web Front-End

(e.g. Apache) Pods

Web Front-End

IngressNginx || HAProxy || etc.

LB Pods

http://*.bikeshop.com

Can be achieved through K8s Ingress or External third Party Load Balancer using NodePort




bution

Nodeint eth0

10.240.0.4

int cbr0

10.24.2.1/24

10.24.2.2 10.24.2.3 10.24.2.4

Kubernetes Networking TopologiesFlat routed topology

ip route 10.24.1.0/24 10.240.0.3

ip route 10.24.2.0/24 10.240.0.4

Nodeint eth0

10.240.0.3

int cbr0

10.24.1.1/24

10.24.1.2 10.24.1.3 10.24.1.4

net.ipv4.ip_forward=1


• Every Node is an IP Router and responsible for its Pod Subnet

• Subnets are associated with Nodes, not Tenants

• Physical Network Configuration is required




bution

Kubernetes Networking TopologiesNode-to-Node overlay topology

Nodeint eth0

10.240.0.4

int cbr0

10.24.2.1/24

10.24.2.2 10.24.2.3 10.24.2.4

Nodeint eth0

10.240.0.3

int cbr0

10.24.1.1/24

10.24.1.2 10.24.1.3 10.24.1.4



Overlay

Key-Value

Store

• Overlays are typically used to avoid Physical Network Configuration




bution

NSX-T and Kubernetes Integration



bution

NSX-T K8s Integration – Namespaces & Pods

admin@k8s-master:~$ kubectl create namespace foonamespace ”foo" created

admin@k8s-master:~$ kubectl create namespace barnamespace ”bar" created

admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foodeployment "nginx-foo" created

admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bardeployment "nginx-bar" created

Namespace: foo Namespace: bar

NSX / K8s topology

10.24.0.0/24 10.24.1.0/24 10.24.2.0/24

NAT boundary

NAT boundary

K8s nodesK8s Masters




bution

NSX-T K8s Integration – Routed Namespaces

admin@k8s-master:~$ vim no-nat-namespace.yaml

apiVersion: v1kind: Namespacemetadata:

name: no-nat-namespaceannotations:

ncp/no_snat: "true“

admin@k8s-master:~$ kubectl create –f no-nat-namespace.yamlnamespace ”no-nat-namespace" created

admin@k8s-master:~$ kubectl run nginx-no-nat --image=nginx –n no-nat-namespacedeployment "nginx-k8s" created

Namespace: no-nat-namespace

NSX / K8s topology

114.4.10.0/26

Direct Routing

114.4.10.64/26

K8s nodesK8s Masters




bution

NSX-T K8s Integration – Pods Micro-SegmentationsOption1: Predefined Label Based Rules

admin@k8s-master:~$ kubectl label pods nginx-foo-3492604561-nltrf secgroup=web -n fooPod "nginx-nsx-3492604561-nltrf" labeled

admin@k8s-master:~$ kubectl label pods nginx-bar-2789337611-z09x2 secgroup=db -n barpod "nginx-k8s-2789337611-z09x2" labeled

admin@k8s-master:~$ kubectl get pods --all-namespaces -LsecgroupNAMESPACE NAME READY STATUS RESTARTS AGE SECGROUPk8s nginx-foo-2789337611-z09x2 1/1 Running 0 58m webnsx nginx-bar-3492604561-nltrf 1/1 Running 0 1h db


NSX / K8s topology

10.24.0.0/24 10.24.1.0/24 114.4.10.0/26

NAT boundary

NAT boundary

Web

• Security Groups are defined in NSX with ingress and egress policy

• Each Security Group could be micro-segmented to protect Pods from each other

DB

21



bution

NSX-T K8s Integration – Pods Micro-SegmentationsOption 2: K8s Network Policy

admin@k8s-master:~$ vim nsx-demo-policy.yamlapiVersion: extensions/v1beta1kind: NetworkPolicymetadata:name: nsx-demo-policy

spec:podSelector:matchLabels:app: web

ingress:- from:

- namespaceSelector:matchLabels:ncp/project: db

ports:- port: 80protocol: TCP

admin@k8s-master:~$ kubectl create -f nsx-demo-policy.yaml

• State: released on K8s 1.7 (Beta on 1.6)

• Capability: Using Network Policy, users can define firewall rules to allow traffic into and out of a Namespace, and between Pods. The network policy is a Namespace property. The default is drop


NSX / K8s topology

10.24.0.0/24 10.24.1.0/24 114.4.10.0/26

NAT boundary

Routed

DB

Label: app=db

Web

Label: app=web




bution

NSX-T K8s Integration – Pods Micro-SegmentationsOption 2: K8s Network Policy

$ kubectl create -f nsx-demo-policy.yaml

23

Dynamic Creation of Security Groups

Dynamic Creation of Security Policy based on k8s Network Policy

Once the Network Policy is applied, NSX will dynamically create source & destination Security Groups and apply the right policy



bution

NSX-T K8s Integration – Pods Micro-Segmentations

• Micro-Segmentation in K8s: The data model to describe segmentation policies between Namespaces, and within namespaces is called ’Network Policies’ and is released on Kubernetes 1.7 (Beta on 1.6)

Firewalling in Kubernetes

• NSX could utilize K8s Network Policies to define Dynamic Security Groups & Policies.

• Capabilities are limited to K8s Network Policy capabilities.

K8s Network Policy

• Security Groups & Policies could be predefined on NSX. Labels are used to specify Pods Membership

• Mapping of IP based groups, egress rules, VM based matching could be available to be used in the policy definition

Pre-Defined Label based rules

• The NSX / K8s integration intends to support both the pre-defined label based rules and K8s network policy.

Firewalling in NSX / K8s




bution

East-West Load Balancing

Node

VM

NSX CNI

Plugin

OVS

Pods

NSX KubeProxy

K8s masterK8s master

K8s

Master

Controller

Manager

K8s API

Server

dashboard

Scheduler

• K8s Services are delivered through NSX Kube-Proxy.

• Delivered as a container image, so that it can be run as a Kubernetes Daemon-Set on the Nodes.

• NSX Kube-Proxy would replace the native distributed east-west load balancer in Kubernetes called Kube-Proxy.

• OpenVSwitch (OVS) load-balancing is used.




bution

North-South Load Balancing

• Once an Ingress Controller is added, NSX will define SNAT & DNAT rules

26

Web Front-End

Ingress

Nginx

Ingress LB Pod

http://*.demo.corp.local

10.4.0.0/24 10.4.1.0/24

10.4.0.67



bution

K8s / NSX ComponentsNSX Container Plugin (NCP)

• NCP is a software component provided by VMware in form of a container image, e.g. to be run as a K8s Pod.

• NCP is build in a modular way, so that individual adapters can be added for different CaaS and PaaS systems




bution

Namespace creation workflowK8s / NSX WorkflowsNamespace / Topology creation

NCP

Infra

K8s

Adapter

NSX Container Plugin

NSX

Manager

API Client

NSX

Manager

NS: foo

NSX/ K8s topology

NS: bar

K8s master

etcd

API-

Server

Scheduler

1)2)

3)

4)

1. NCP creates a ‚watch‘ on K8s API for any Namespace events

2. A user creates a new K8s Namespace

3. The K8s API Server notifies NCP of the change (addition) of Namespaces

4. NCP creates the network topology for the Namespace :

a) Requests a new subnet from the pre-configured IP block in NSX

b) Creates a logical switchc) Creates a T1 router and attaches it to

the pre-configured global T0 routerd) Creates a router port on the T1 router,

attaches it to the LS, and assigns an IP from the new subnet




bution

NSX-T Container Interface (CIF)

Hypervisor

(ESXi &

KVM)

Node

VM

DFW

eth2

Node

VM

DFW

eth0

Minion Mgmt.

IP Stack

eth0

Minion Mgmt.

IP Stack

mgmtnetwork

OVS

mgmtnetwork

Vla

n10

vla

n11

cifcif

eth2

vla

n10

vla

n11

OVS

cifcif

NSX CNI

Plugin

NSX CNI

Plugin

Pods

Pods

29

• Management Interface is Separated

from the interface used for Pods traffic

• CIF is used per K8s Pod

• CIFs are differentiated through locally

significant VLAN tags

• NSX CNI Plugin is responsible for

tagging the traffic with the right VLAN

• NCP will map the VLAN tags to a

specific CIF.




bution

NSX-T Operational Tools for K8s

30

NSX-T Traceflow

NSX-T Operational Tools• Traceflow• Port Mirroring• Port Connection Tool• Spoofguard• Syslog• Port Counters• IPFIX




bution

Demo



bution

NSX-T Values for K8s

Enterprise-class

Networking

Advanced Security Enhanced

Operations

Full Network

Visibility

Enterprise

Support

Unified VM-to-

Pod

Networking

Pods Micro-

Segmentation

N S X - T Va l u e s f o r K 8 s

F e a t u r e s




bution

33

Hands On Lab Self-Paced Lab

VMware NSX-T with KubernetesSPL182602U(US)

SPL182602E(Europe)

Kubernetes and VMware NSX Blog

https://blogs.vmware.com/networkvirtualization/2017/03/

kubecon-2017.html/



bution

https://blogs.vmware.com/networkvirtualization/2017/03/kubecon-2017.html/

Join VMUG for exclusive access to NSX

vmug.com/VMUG-Join/VMUG-Advantage

Connect with your peers

communities.vmware.com

Find NSX Resources

vmware.com/products/nsx

Network Virtualization Blog

blogs.vmware.com/networkvirtualization

Where to get started

Dozens of Unique NSX Sessions

Spotlights, breakouts, quick talks & group discussions

Visit the VMware Booth

Product overview, use-case demos

Visit Technical Partner Booths

Integration demos – Infrastructure, security, operations,

visibility, and more

Meet the Experts

Join our Experts in an intimate roundtable discussion

Free Hands-on Labs

Test drive NSX yourself with expert-led or self-paces

hands-on labs

labs.hol.vmware.com

Training and Certification

Several paths to professional certifications. Learn

more at the Education & Certification Lounge.

vmware.com/go/nsxtraining

Engage and Learn Experience

Try TakeVMworld 2017 Content: N

ot for publicatio

n or distribution

https://www.vmug.com/VMUG-Join/VMUG-Advantage

https://communities.vmware.com/community/vmtn/nsx

http://vmware.com/products/nsx

http://blogs.vmware.com/networkvirtualization

http://labs.hol.vmware.com/

http://www.vmware.com/go/nsxtraining



bution

NET1522BE Kubernetes Networking with NSX-T … Al Idrees Yves Fauser NET1522BE #VMworld #NET1522BE...

Documents

Transcript of NET1522BE Kubernetes Networking with NSX-T … Al Idrees Yves Fauser NET1522BE #VMworld #NET1522BE...