Kubernetes Container Networking with NSX-T Data …...Kubernetes Container Networking with NSX-T...

#vmworld

KubernetesContainer Networking with

NSX-T Data CenterDeep Dive

Yasen Simeonov, Sr. Technical Product Manager, NSBUDennis Breithaupt, Sr. Systems Engineer (NSX)

,

NET1677BE

#NET1677BE

VMworld 2018 Content: Not for publication or distribution

Disclaimer

2©2018 VMware, Inc.

This presentation may contain product features orfunctionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.


Agenda


NSX-T IntroQuick level set on NSX-T

Kubernetes OverviewTechnical overview of Kubernetes, nomenclature & networking details

NSX-T & KubernetesDetails of the NSX-T integration with Kubernetes

DemoSeeing is believing



NSX-T Data Center IntroQuick level set on NSX-T Data Center



BRANCH

BRANCH

BRANCH

BRANCH

BRANCH

BRANCH

BRANCH

BRANCH

TELCO/NFV

TELCO/NFV

EDGE/IOT

TELCO/NFV

BRANCH

BRANCH

EDGE/IOT

EDGE/IOT

The Virtual Cloud NetworkConnect and Protect your BusinessVMworld 2018 Content: Not for publication or distribution


Identity

Apps and Data

Policy ScalabilityAnalytics and Insights

Secure Connectivity Availability

Users

Private Data Centers

VMs, Containers, Microservices

Branch Offices

Public Clouds

Telco Networks

Things

Virtual Cloud NetworkingConnect & Protectany workload across any environment

Built-in

Automated

Programmable

Application Centric



NETWORKING AND SECURITY MANAGEMENT AND AUTOMATION

vRealize AutomationEnd-to-end workload automation

Network InsightNetwork discovery and insights

Cloud-Based Management Workflow Automation Blueprints / Templates Insights / Discovery Visibility

NETWORK AND SECURITY VIRTUALIZATION

AppDefenseModern application

security

NSX SD-WANby VeloCloud

WAN connectivity services

NSX Hybrid ConnectData center and cloud

workload migration

NSX Data CenterNetworking and

security for data center workloads

NSX CloudNetworking and

security for Public Cloud workloads

Security Integration Extensibility Automation Elasticity

VMware NSX PortfolioThe Foundation of the Virtual Cloud Network



NSX Data Center Architecture For Private Cloud, Public Cloud & Containers

CONTROLPLANE

DATAPLANE

MANAGEMENT PLANE

Private or Public cloud infrastructure

NSX Central Controller

NSX Manager

(VPN Gateway, DirectConnect, ExpressRoute)

Public Cloud

Linux VM Windows VMNSX Cloud Gateway

VMware Cloud on AWS

Private Cloud

NSX Edge VM or Bare Metal

ESXi KVM

N-VDS N-VDS

Multi-Hypervisor

Cloud ServiceManager

Bare Metal

NSX NSX



Data PlaneImproved performance and resiliency

Admin

Tenants/CMP

Designed for multi-tenancy and scale

New distributed edge architecture with increased

performance with DPDK

p1 p2

HV TN1vSwitch1

TEP

Overlay Transport Zone

TEP: Overlay Tunnel End Point

(with its own IP address)

GENEVE Tunnel

p1 p2

HV TN1 vSwitch2

TEP

Next gen overlay maintaining

performance with increased flexibility

EdgeNode

Edge Cluster

EdgeNode

EdgeNode

EdgeNode

Admin

Tenants/CMP

Designed for multi-tenancy and scale

SessionsNET1127BU VMware NSX-T™ Data Center Routing Deep Dive



Kubernetes OverviewTechnical overview of Kubernetes, nomenclature & networking details



Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts, providing container-centric infrastructure.

What is Kubernetes?



Kubernetes Components

K8s Cluster Consists of Master(s) and Nodes

K8s Master Components• API Server• Scheduler• Controller Manager• Dashboard

K8s Node Components• Kubelet• Kube-Proxy• Containers Runtime

K8s masterK8s master

K8s Master

Controller Manager

K8s APIServer

Key-Value Store

dashboard

Scheduler

K8s nodeK8s nodeK8s nodeK8s node

K8s Nodes

kubelet c runtime

Kube-proxy

> _ Kubectl

CLI

K8s Master(s)



Kubernetes Pod

A Pod is a group of one or more containers that shares an IP address and a Data Volume Pod

pause container(‘owns’ the IP stack)

10.24.0.0/16

10.24.0.2

nginxtcp/80

mgmttcp/22

loggingudp/514

IPC

External IP Traffic



Kubernetes Namespace

Namespaces are a way to divide cluster resources amongst users and groups

They can be thought of as Tenants

They are a way to provide Resources Quotas, RBAC, Networking Multitenancy, and Name uniqueness

Namespace: fooBase URI: /api/v1/namespaces/foo

‚redis-master‘ Pod:/api/v1/namespaces/foo/pods/redis-master

‚redis‘ service:/api/v1/namespaces/foo/services/redis-master

Namespace: barBase URI: /api/v1/namespaces/bar

‚redis-master‘ Pod:/api/v1/namespaces/bar/pods/redis-master

‚redis‘ service:/api/v1/namespaces/bar/services/redis-master



Kubernetes Service

A Kubernetes Service defines a logical set of Pods, selected with matching labels

Serves multiple functions:• Service Discovery / DNS• East/West load balancing in

the Cluster (Type: ClusterIP)

• External load balancing for L4 TCP/UDP (Type: LoadBalancer)

• External access to the service through the nodes IPs (Type: NodePort)

Redis Slave Pods

redis-slave svc

10.24.0.5

ClusterIP172.30.0.24

Web Front-EndPods

10.24.2.7

▶ kubectl describe svc redis-slaveName: redis-slaveNamespace: defaultLabels: name=redis-slaveSelector: name=redis-slaveType: LoadBalancerIP: 172.30.0.24LoadBalancer Ingress: 134.247.200.20Port: <unnamed> 6379/TCPEndpoints: 10.24.0.5:6379,

10.24.2.7:6379

DNS:

redis-slave.<ns>.cluster.local 172.30.0.24

ExternalIP134.247.200.20

DNS:

redis-slave.external.com 134.247.200.20



Kubernetes Ingress

A Kubernetes Ingress Object is a L7 LoadBalancing rule that binds a hostname and url to a Service

The LoadBalancer Datapath can be implemented as an external Load Balancer or as a K8s Pod

Web Front-EndPods (shop svc)

http://www.bikeshop.com/shop

Web Front-EndPods (special-offers svc)

http://www.bikeshop.com/special-offers

LoadBalancer Datapath

(External or K8s Pods)

▶ kubectl describe ingress bikeshop-ingress-shopName: bikeshop-shopNamespace: bikeshopAddress: 100.64.240.9,134.247.200.1Default backend: default-http-backend:80 (<none>)

Rules:Host Path Backends---- ---- --------www.bikeshop.com /shop

web-svc-1:80 (<none>)

External IP: 134.247.200.1

DNS: *.bikeshop.com 134.247.200.1



Kubernetes Networking Topologies

Every Node is an IP Router and responsible for its Pod Subnet

Subnets are associated with Nodes, not Tenants

Physical Network Configuration is required

Non-multitenant routed topology

Nodeint eth0

10.240.0.4

int cbr0

10.24.2.1/24

10.24.2.2 10.24.2.3 10.24.2.4

ip route 10.24.1.0/24 10.240.0.3ip route 10.24.2.0/24 10.240.0.4

Nodeint eth0

10.240.0.3

int cbr0

10.24.1.1/24

10.24.1.2 10.24.1.3 10.24.1.4

net.ipv4.ip_forward=1




Kubernetes Networking Topologies

Overlays are typically used to avoid Physical Network Configuration

Subnets are still associated with Nodes, not Tenants

External outbound connectivity needs SNAT using the Nodes IP

External inbound connectivity needs Node Port or Ingress in Host Network Mode

Node-to-Node overlay topology

Nodeint eth0

10.240.0.4

int cbr0

10.24.2.1/24

10.24.2.2 10.24.2.3 10.24.2.4

Nodeint eth0

10.240.0.3

int cbr0

10.24.1.1/24

10.24.1.2 10.24.1.3 10.24.1.4



Overlay

Key-Value Store



NSX-T Data Center & KubernetesDetails of the NSX-T Data Center integration with Kubernetes



Key design goals of the NSX-T Data Center Kubernetes Integration

Don't stand in the way of the developer !

Provide solutions to map the Kubernetes

constructs to enterprise networking

constructs

Secure Containers, VMs and any other

endpoints with overarching Firewall

Policies

Provide visibility & troubleshooting tools to ease the

container adoption in the enterprise


NSX-T K8s Integration – Namespaces & Pods

admin@k8s-master:~$ kubectl create namespace foonamespace ”foo" created

admin@k8s-master:~$ kubectl create namespace barnamespace ”bar" created

admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foodeployment "nginx-foo" created

admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bardeployment "nginx-bar" created

Namespace: foo Namespace: bar

NSX / K8s topology

10.24.0.0/24 10.24.1.0/24

10.24.2.0/24

NAT boundary

NAT boundary

K8s nodesK8s Masters

Dynamic per Namespace Topology


NSX-T K8s Integration – Routed Namespaces

admin@k8s-master:~$ vim no-nat-namespace.yaml

apiVersion: v1kind: Namespacemetadata:

name: no-nat-namespaceannotations:

ncp/no_snat: "true“

admin@k8s-master:~$ kubectl create –f no-nat-namespace.yamlnamespace ”no-nat-namespace" created

admin@k8s-master:~$ kubectl run nginx-no-nat --image=nginx –n no-nat-namespacedeployment "nginx-k8s" created

Namespace: no-nat-namespace

NSX / K8s topology

114.4.10.0/26

Direct Routing

114.4.10.64/26

K8s nodesK8s MastersVMworld 2018 Content: Not for publication or distribution


K8s / NSX-T Data Center Components

NCP is a software component provided by VMware in form of a container image, e.g. to be run as a K8s Pod.

NCP is build in a modular way, so that individual adapters can be added for different CaaS and PaaS systems at some point

NSX Container Plugin (NCP)

NCM Infra

K8s / OSAdapter

CloudFoundryAdapter

NSX Container Plugin

More…

NSX Manager

API Client

NSX Manager

NS: foo NS: barNSX/ K8s topology

K8s master

etcd

API-Server

Scheduler



Tenancy / Topology MappingThe open source way

With most networking technologies in K8s the source IP of the traffic can't be mapped to the tenancy. This is the biggest hurdle today to get K8s integrated in enterprise IT environments

Node VM

IPTables(NAT)

vnic

mgmt IP

Pods

10.255.0.10/2410.255.0.9/24

172.16.1.11/24

Node VM

IPTables(NAT)

vnic

mgmt IP

Pods

10.255.1.3/2410.255.1.5/24

172.16.1.12/24

Physical or virtual Router

172.16.1.1/24

Tenant: fooTenant: barTenant: foo

Database (VM based or Physical)

Physical DC FirewallSNAT to Node IP

Did the traffic come from 'foo' or 'bar'?

SNAT to Node IP



Tenancy / Topology MappingPersistent IPs for K8s Namespaces

With NSX-T each Tenant (Kubernetes Namespace) either gets its own SNAT IP (NAT Mode), or is directly identifiable by its source subnet (No NAT Mode)

Node VM

OpenvSwitch

10.12.5.5/2410.12.1.8/24

172.16.1.11/24

mgmt IP

vnic

Namesp. FooT1 router

PAS VMsT1 router

VLAN Trunk

NSX-T Logical Switch

Namesp. BarT1 router

172.16.1.1/24 10.12.1.1/24 10.12.5.1/24

Pods

Database (VM based or Physical)

Physical DC Firewall

A new SNAT IP is allocated on the T0 router for each Tenant for NAT Mode

In NAT Mode, the external DC Firewall and the DB can distinguish tenant 'foo' and tenant 'bar' using the source SNAT IP that is allocated to a specific Tenant.

Tenant: fooTenant: bar

In No-NAT Mode, the external DC Firewall and the DB can distinguish tenant 'foo' and tenant 'bar' using the source IP Subnet that is allocated to a specific Tenant.



Infrastructure Teams can pre-create Firewall rules in existing DC physical Firewalls to allow traffic from specific workloads in K8s

The K8s user / DevOps can deploy applications that are easily identifiable in the physical network

With this feature a set of Kubernetes Workloads (Pods) can be assigned to use a specific IP or group of SNAT IPs to source their traffic from

Before this feature only a SNAT IP to a Kubernetes Namespace was assigned

Feature

Benefits

Persistent SNAT IP per K8s ServiceSpecifying the source IP Kubernetes Workloads using the K8s service

Tier0 LR

Corporate network

DB

allow – from: 134.247.100.10 (App) to: 134.247.200.9 (DB)

Tier1 LR

Kubernetes Namespace: Foo

Web-FrontendPods

App Logic Pods

K8s Svc for AppK8s Svc for Web

Namespace LS(s)

SNAT App Svc Pods to: 134.247.100.10For all other Pods

use namespace SNAT IP


Central Visibility

With NSX-T you have deep visibility and troubleshooting tools.


Kubernetes Metadata / NSX Logical Port Mapping

▶ kubectl get pod nsx-demo-rc-c7x65 -o yaml

apiVersion: v1kind: Podmetadata:creationTimestamp: 2018-07-25T12:05:56ZgenerateName: nsx-demo-rc-labels:

app: nsx-demoname: nsx-demo-rc-c7x65namespace: nsx-ujo

Metadata within Kubernetes like Namespace, Pod names, Labels all get copied to the NSX Logical Port as Port Tags


Pre-Created Security Groups / Firewall rules (admin rules)NSX can be configured to collect ports and switches in dynamic security groups based on Tags (Kubernetes Metadata) and apply Firewall rules on them

Match on Port Tags

Matching Pods are part of the Group

Groups are used in Firewall sections as src and dst


Support of Kubernetes Network Policy

Besides supporting admin pre-defined rules, NCP is also translating Kubernetes NetworkPolicy Objects to NSX security groups and Firewall rules

Admin pre-defined rules can be used concurrently in NSX, admin rules are put in sections before or after K8s network policy rules

apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:name: default-deny

spec:podSelector: {}policyTypes:- Ingress

apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:

name: nsx-demo-policyspec:

podSelector:matchLabels:

app: nsx-demopolicyTypes:- Ingressingress:- from:

- ipBlock:cidr: 100.64.160.11/32

ports:- port: 80

protocol: TCP


Built-in Load Balancing

NCM Infra

K8s / OSAdapter

CloudFoundry Adapter

Libnetwork Adapter


More…

NSX Manager

API Client

NSX Manager

K8s master

etcd

API-Server

Scheduler

Virtual Server10.114.209.209HTTP and/or

HTTPS traffic

Server Pool 1

Server Pool 2Rule 2/bar/

Rule 1/foo/

LB Service

NCM Infra

K8s / OSAdapter

CloudFoundry Adapter

Libnetwork Adapter


More…

NSX Manager

API Client

NSX Manager

K8s master

etcd

API-Server

Scheduler

Virtual Server10.114.209.212TCP and/or

UDP traffic

Server Pool

LB Service

Built-in support for Ingress (L7 HTTP/HTTPS) and Svc Type LB (L4 TCP/UDP) in the NSX-K8s integration. Most other K8s networking choice don't support Svc Type LB (L4), and you need an additional technology like NGINX from Ingress (L7).


K8s NSX Plugin – Child Interfaces

• CIFs (Child VIFs) are subinterfaces of parent VIFs (vnic/vm interfaces)

• The OVS on the Worker VM tags traffic coming from a POD with a local significant VLAN id

• NSX vSwitch on the Hypervisor have a logical (sub-interface) and is able to apply DFW rules to specific CIFs

• The OVS on the Worker is ‘standalone’, it is not controlled by the NSX control plane and is only programed by the NSX CNI Plugin

Hypervisor(ESXi &

KVM)

K8s WorkerVM

Pod

DFW

eth3

Pod

T1 Router

K8s WorkerVM

Pod

DFW

Pod

eth0

Minion Mgmt. IP Stack

eth0

Minion Mgmt. IP Stack

mgmtnetwork

OVS

mgmtnetwork

Vla

n1

vlan

2

cifcif

eth3

vlan

1

vlan

2

OVS

cifcif

NSX CNI

Plugin

NSX CNI

Plugin


Relational database used to store FAA registration relational data, e.g., Planes => Planetypes

No-SQL in-memory database used to store

high-churn Aircraft position data

Receives lives aircraft position data and feeds it

into redis no-sql cache

Automatic dependent surveillance broadcast

(ADS–B) - live feed(ADS–B)

Provides a REST API to frontend, takes data from MySQL and redis and correlates information

Python Flask and Bootstrap based Web-Frontent

http (web)

http (REST)TCP stream

RESP

RESP

MySQL

Application Overview

Planespotter: retrieve FAA aircraft registration data and airborne status

API App ServiceADS-B data feeder

Frontend

Thanks to Yves Fauser


NSX-T Data Center Values for Containers

Enterprise-class Networking

Advanced Security

Enhanced Operations

Full Network Visibility

Enterprise Support

Unified VM-to-Container Networking

Micro-Segmentation

N S X - T V a l u e s f o r C o n t a i n e r s

F e a t u r e sVMworld 2018 Content: Not for publication or distribution


Join the NSX VMUG Communityvmug.com/nsxConnect with your Peerscommunities.vmware.com

Embrace the NSX Mindsetnsxmindset.comFind NSX Resourcesvmware.com/go/networkingRead the Network Virtualization Blogblogs.vmware.com/networkvirtualization

Where to get started

Attend the Networking and Security SessionsShowcases, breakouts, quick talks & group discussions

Visit the VMware BoothProduct overviews, use-case demos

Visit Technical Partner BoothsIntegration demos – Infrastructure, security, operations, visibility, and more

Meet the ExpertsJoin our experts in an intimate roundtable discussion

Free Hands-on Labslabs.hol.vmware.com

Virtual Cloud Network Guided Demovcndemo.com

VMware Education - Training and Certificationvmware.com/go/nsxtraining

Free NSX Training on Courseravmware.com/go/coursera

Engage and Learn Experience

Try Take


http://vmug.com/nsx

https://communities.vmware.com/community/vmtn/nsx

http://www.vmwarensxmindset.com/

http://www.vmware.com/go/networking

http://blogs.vmware.com/networkvirtualization

http://labs.hol.vmware.com/

http://www.vcndemo.com/

http://www.vmware.com/go/nsxtraining

http://www.vmware.com/go/coursera

DON’T FORGET TO FILL OUT YOUR SURVEY.

#vmworld #NET1677BE


THANK YOU!

#vmworld #NET1677BE


Kubernetes Container Networking with NSX-T Data …...Kubernetes Container Networking with NSX-T...

Documents

Transcript of Kubernetes Container Networking with NSX-T Data …...Kubernetes Container Networking with NSX-T...