or distribution Networking with NSX-T for publication Kubernetes Container · 2018-09-05 ·...

#vmworld

Kubernetes Container Networking with NSX-TData Center Deep Dive

Yasen Simeonov, VMware, Inc.

NET1677BU

#NET1677BU

VMworld 2018 Content: Not for publication or distribution

Disclaimer

2©2018 VMware, Inc.

This presentation may contain product features orfunctionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.


Agenda


NSX-T IntroQuick level set on NSX-T

Kubernetes OverviewTechnical overview of Kubernetes, nomenclature & networking details

NSX-T & KubernetesDetails of the NSX-T integration with Kubernetes

DemoSeeing is believing



NSX-T Data Center IntroQuick level set on NSX-T Data Center



BRANCH

BRANCH

BRANCH

BRANCH

BRANCH

BRANCH

BRANCH

BRANCH

TELCO/NFV

TELCO/NFV

EDGE/IOT

TELCO/NFV

BRANCH

BRANCH

EDGE/IOT

EDGE/IOT

The Virtual Cloud NetworkConnect and protect your business



Identity

Apps and Data

Policy ScalabilityAnalytics and Insights

Secure Connectivity Availability

Users

Private Data Centers

VMs, Containers, Microservices

Branch Offices

Public Clouds

Telco Networks

Things

Virtual Cloud NetworkingConnect & Protectany workload across any environment

Built-in

Automated

Programmable

Application Centric



NETWORKING AND SECURITY MANAGEMENT AND AUTOMATION

vRealize AutomationEnd-to-end workload automation

Network InsightNetwork discovery and insights

Cloud-Based Management Workflow Automation Blueprints / Templates Insights / Discovery Visibility

NETWORK AND SECURITY VIRTUALIZATION

AppDefenseModern application

security

NSX SD-WANby VeloCloud

WAN connectivity services

NSX Hybrid ConnectData center and cloud

workload migration

NSX Data CenterNetworking and

security for data centerworkloads

NSX CloudNetworking and

security for Public Cloud workloads

Security Integration Extensibility Automation Elasticity

VMware NSX PortfolioThe foundation of the Virtual Cloud Network



Central Control ClusterCCP

Local Control PlaneLCP

NSX-T Data Center Architecture and Components

Cloud Consumption

Data PlaneESXi

(+ kernel modules)

Control Plane

Management Plane

NSX Edge VM or

Bare Metal

Layer 2 Bridge

KVM(+ kernel modules)

Highly available and scalableBuilt for consumption by

developers

Support for endpoint heterogeneity

Improved performance and resiliency

OpenStack, k8s or Custom



Data Plane

Improved performance and resiliency

Admin

Tenants/CMP

Designed for multi-tenancy and scale

New distributed edge architecture with increased

performance with DPDK

p1 p2

HV TN1vSwitch1

TEP

Overlay Transport Zone

TEP: Overlay Tunnel End Point

(with its own IP address)

GENEVE Tunnel

p1 p2

HV TN1 vSwitch2

TEP

Next gen overlay maintaining

performance with increased flexibility

EdgeNode

Edge Cluster

EdgeNode

EdgeNode

EdgeNode



Kubernetes OverviewTechnical overview of Kubernetes, nomenclature & networking details



Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts, providing container-centric infrastructure.

What is Kubernetes?



Kubernetes Components

K8s Cluster Consists of Master(s) and Nodes

K8s Master Components• API Server• Scheduler• Controller Manager• Dashboard

K8s Node Components• Kubelet• Kube-Proxy• Containers Runtime

(Docker or Rocket)

K8s masterK8s master

K8s Master

Controller Manager

K8s APIServer

Key-Value Store

dashboard

Scheduler

K8s nodeK8s nodeK8s nodeK8s node

K8s Nodes

kubelet c runtime

Kube-proxy

> _ Kubectl

CLI

K8s Master(s)



Kubernetes Pod

A Pod is a group of one or more containers that shares an IP address and a Data Volume Pod

pause container(‘owns’ the IP stack)

10.24.0.0/16

10.24.0.2

nginxtcp/80

mgmttcp/22

loggingudp/514

IPC

External IP Traffic



Kubernetes Namespace

Namespaces are a way to divide cluster resources amongst users and groups

They can be thought of as Tenants

They are a way to provide Resources Quotas, RBAC, Networking Multitenancy, and Name uniqueness

Namespace: fooBase URI: /api/v1/namespaces/foo

‚redis-master‘ Pod:/api/v1/namespaces/foo/pods/redis-master

‚redis‘ service:/api/v1/namespaces/foo/services/redis-master

Namespace: barBase URI: /api/v1/namespaces/bar

‚redis-master‘ Pod:/api/v1/namespaces/bar/pods/redis-master

‚redis‘ service:/api/v1/namespaces/bar/services/redis-master



Kubernetes Service

A Kubernetes Service defines a logical set of Pods, selected with matching labels

Serves multiple functions:• Service Discovery / DNS• East/West load balancing

in the Cluster (Type: ClusterIP)

• External load balancing for L4 TCP/UDP (Type: LoadBalancer)

• External access to the service through the nodes IPs (Type: NodePort)

Redis Slave Pods

redis-slave svc

10.24.0.5

ClusterIP172.30.0.24

Web Front-EndPods

10.24.2.7

▶ kubectl describe svc redis-slaveName: redis-slaveNamespace: defaultLabels: name=redis-slaveSelector: name=redis-slaveType: LoadBalancerIP: 172.30.0.24LoadBalancer Ingress: 134.247.200.20Port: <unnamed> 6379/TCPEndpoints: 10.24.0.5:6379,

10.24.2.7:6379

DNS:

redis-slave.<ns>.cluster.local 172.30.0.24

ExternalIP134.247.200.20

DNS:

redis-slave.external.com 134.247.200.20



Kubernetes Ingress

A Kubernetes Ingress Object is a L7 LoadBalancing rule that binds a hostname and url to a Service

The LoadBalancer Datapath can be implemented as an external Load Balancer or as a K8s Pod

Web Front-EndPods (shop svc)

http://www.bikeshop.com/shop

Web Front-EndPods (special-offers svc)

http://www.bikeshop.com/special-offers

LoadBalancer Datapath

(External or K8s Pods)

▶ kubectl describe ingress bikeshop-ingress-shopName: bikeshop-shopNamespace: bikeshopAddress: 100.64.240.9,134.247.200.1Default backend: default-http-backend:80 (<none>)

Rules:Host Path Backends---- ---- --------www.bikeshop.com /shop

web-svc-1:80 (<none>)

External IP: 134.247.200.1

DNS: *.bikeshop.com 134.247.200.1



Kubernetes Networking Topologies

Every Node is an IP Router and responsible for its Pod Subnet

Subnets are associated with Nodes, not Tenants

Physical Network Configuration is required

Non-multitenant routed topology

Nodeint eth0

10.240.0.4

int cbr0

10.24.2.1/24

10.24.2.2 10.24.2.3 10.24.2.4

ip route 10.24.1.0/24 10.240.0.3ip route 10.24.2.0/24 10.240.0.4

Nodeint eth0

10.240.0.3

int cbr0

10.24.1.1/24

10.24.1.2 10.24.1.3 10.24.1.4

net.ipv4.ip_forward=1




Kubernetes Networking Topologies

Overlays are typically used to avoid Physical Network Configuration

Subnets are still associated with Nodes, not Tenants

External outbound connectivity needs SNAT using the Nodes IP

External inbound connectivity needs Node Port or Ingress in Host Network Mode

Node-to-Node overlay topology

Nodeint eth0

10.240.0.4

int cbr0

10.24.2.1/24

10.24.2.2 10.24.2.3 10.24.2.4

Nodeint eth0

10.240.0.3

int cbr0

10.24.1.1/24

10.24.1.2 10.24.1.3 10.24.1.4



Overlay

Key-Value Store



NSX-T & KubernetesDetails of the NSX-T integration with Kubernetes



Key Design Goals of the NSX-T Data Center Kubernetes Integration

Don't stand in the way of the developer!

Provide solutions to map the Kubernetes

constructs to enterprise networking

constructs

Secure Containers, VMs and any other

endpoints with overarching Firewall

Policies

Provide visibility & troubleshooting tools to ease the

container adoption in the enterprise



Kubernetes NSX Topology

Dynamically network topology per K8s namespace

K8s Nodes are not doing IP routing

Every Pod has its own logical port on a NSX logical switch, and is supporting all features a VM interface supports

Every Pod has Dynamic Firewall rules applied on its logical Interface

Dynamic per Namespace Topology

Namespace: foo Namespace: bar

NSX/ K8s topology

10.4.0.0/26 10.4.0.64/26 34.1.2.33/26



K8s / NSX Components

NCP is a software component provided by VMware in form of a container image, e.g. to be run as a K8s Pod.

NCP is build in a modular way, so that individual adapters can be added for different CaaS and PaaS systems at some point

NSX Container Plugin (NCP)

NCM Infra

K8s / OSAdapter

CloudFoundryAdapter

NSX Container Plugin

More…

NSX Manager

API Client

NSX Manager

NS: foo NS: barNSX/ K8s topology

K8s master

etcd

API-Server

Scheduler



With most networking technologies in K8s like Flannel, OpenShift OVS Networking, Calico, etc. the source IP of the traffic can't be mapped to the tenancy. This is the biggest hurdle today to get K8s integrated in enterprise IT environments

Tenancy / Topology MappingThe open source way

Node VM

IPTables(NAT)

vnic

mgmt IP

Pods

10.255.0.10/2410.255.0.9/24

172.16.1.11/24

Node VM

IPTables(NAT)

vnic

mgmt IP

Pods

10.255.1.3/2410.255.1.5/24

172.16.1.12/24

Physical or virtual Router

172.16.1.1/24

Tenant: fooTenant: barTenant: foo

Database (VM based or Physical)

Physical DC FirewallSNAT to Node IP

Did the traffic come from 'foo’ or 'bar'?

SNAT to Node IP



With NSX-T each Tenant (Kubernetes Namespace) either gets its own SNAT IP (NAT Mode), or is directly identifiable by its source subnet (No NAT Mode)

Tenancy / Topology MappingPersistent IPs for K8s Namespaces

Node VM

OpenvSwitch

10.12.5.5/2410.12.1.8/24

172.16.1.11/24

mgmt IP

vnic

Namesp. FooT1 router

PAS VMsT1 router

VLAN Trunk

NSX-T Logical Switch

Namesp. BarT1 router

172.16.1.1/24 10.12.1.1/24 10.12.5.1/24

Pods

Database (VM basedor Physical)

Physical DC Firewall

A new SNAT IP is allocated on the T0 router for each Tenant for NAT Mode

In NAT Mode, the external DC Firewall and the DB can distinguish tenant 'foo' and tenant 'bar' using the source SNAT IP that is allocated to a specific Tenant.

Tenant: fooTenant: bar

In No-NAT Mode, the external DC Firewall and the DB can distinguish tenant 'foo' and tenant 'bar' using the source IP Subnet that is allocated to a specific Tenant.



Infrastructure Teams can pre-create Firewall rules in existing DC physical Firewalls to allow traffic from specific workloads in K8s

The K8s user / DevOps can deploy applications that are easily identifiable in the physical network

With this feature a set of Kubernetes Workloads (Pods) can be assigned to use a specific IP or group of SNAT IPs to source their traffic from

Before this feature we only assigned a SNAT IP to a Kubernetes Namespace

Feature

Benefits

Persistent SNAT IP per K8s ServiceSpecifying the source IP Kubernetes Workloads using the K8s service

Tier0 LR

Corporate network

DB

allow – from: 134.247.100.10 (App) to: 134.247.200.9 (DB)

Tier1 LR

Kubernetes Namespace: Foo

Web-FrontendPods

App Logic Pods

K8s Svc for AppK8s Svc for Web

Namespace LS(s)

SNAT App Svc Pods to: 134.247.100.10For all other Pods

use namespace SNAT IP



Central Visibility

With most other networking technologies in K8s and PCF like Flannel, OpenShift OVS Networking, PCF Silk, Calico, etc. there is no centralized control plane. So, there’s no counters, troubleshooting tools, 'span ports', Firewall Rules Overview, etc.



Central Visibility

With NSX-T you are gain deep visibility into the container networks, and you can use the same troubleshooting tools we created for VM based workloads



Kubernetes Metadata / NSX Logical Port Mapping

▶ kubectl get pod nsx-demo-rc-c7x65 -o yaml

apiVersion: v1kind: Podmetadata:creationTimestamp: 2018-07-25T12:05:56ZgenerateName: nsx-demo-rc-labels:

app: nsx-demoname: nsx-demo-rc-c7x65namespace: nsx-ujo

Metadata within Kubernetes like Namespace, Pod names, Labels all get copied to the NSX Logical Port as Port Tags



Pre-Created Security Groups / Firewall rules (admin rules)

NSX can be configured to collect ports and switches in dynamic security groups based on Tags (Kubernetes Metadata) and apply Firewall rules on them

Match on Port Tags

Matching Pods are part of the Group

Groups are used in Firewall sections as src and dst



Unified Policy for K8s, PCF & VMs

Both K8s and PCF have 'built-in' micro segmentation policy languages (network policy), and there's a broad set of products and open source projects implementing micro segmentation inside of K8s or PCF. However there is no technology other than NSX-T today that allows you to define policies across K8s, PCF and VM based workloads using Metadata from each system

PCF Org FooT1 routerDB VMs

T1 router

Kubenetes Namespace: BarT1 router

NSX-T Logical Switch NSX-T LS NSX-T Logical Switch

K8s PodsPCF AIs

vSphere VMs

allow: tcp/443

allow: tcp/3306 (mysql)



Support of Kubernetes Network Policy

Besides supporting admin pre-defined rules, NCP is also translating Kubernetes NetworkPolicy Objects to NSX security groups and Firewall rules

Admin pre-defined rules can be used concurrently in NSX, admin rules are put in sections before or after K8s network policy rules

apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:name: default-deny

spec:podSelector: {}policyTypes:- Ingress

apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:

name: nsx-demo-policyspec:

podSelector:matchLabels:

app: nsx-demopolicyTypes:- Ingressingress:- from:

- ipBlock:cidr: 100.64.160.11/32

ports:- port: 80

protocol: TCP



Built-in Load Balancing

NCM Infra

K8s / OSAdapter

CloudFoundry Adapter

Libnetwork Adapter


More…

NSX Manager

API Client

NSX Manager

K8s master

etcd

API-Server

Scheduler

Virtual Server10.114.209.209HTTP and/or

HTTPS traffic

Server Pool 1

Server Pool 2Rule 2/bar/

Rule 1/foo/

LB Service

NCM Infra

K8s / OSAdapter

CloudFoundry Adapter

Libnetwork Adapter


More…

NSX Manager

API Client

NSX Manager

K8s master

etcd

API-Server

Scheduler

Virtual Server10.114.209.212TCP and/or

UDP traffic

Server Pool

LB Service

We have built-in support for Ingress (L7 HTTP/HTTPS) and Svc Type LB (L4 TCP/UDP) in the NSX-K8s integration. Most other K8s networking choice don't support Svc Type LB (L4), and you need an additional technology like NGINX from Ingress (L7).



K8s / NSX Workflows

1. NCP watches for Svc events in Kubernetes

2. User creates a new Svc of Type LoadBalancer

3. The Kubernetes API server notifies NCP of the new Svc

4. NCP creates a new Virtual Server with a unique IP and a Server Pool with the Pods as targets

Svc Type LB

NCM Infra

K8s / OSAdapter

CloudFoundryAdapter


More…

NSX Manager

API Client

NSX Manager

K8s master

etcd

API-Server

Scheduler

1)2)

3)

4)

Virtual Server10.114.209.212TCP

and/orUDP traffic

Server Pool

LB Service



K8s / NSX Workflows

1. NCP watches for Ingress events in Kubernetes

2. User creates a new Ingress rule

3. The Kubernetes API server notifies NCP of the new Ingress rule

4. NCP creates a new forwarding rule sending a specific HTTP/S hostname and path to a specific Server Pool

Ingress

NCM Infra

K8s / OSAdapter

CloudFoundryAdapter


More…

NSX Manager

API Client

NSX Manager

K8s master

etcd

API-Server

Scheduler

1)2)

3)

4)

LB Service

Virtual Server 10.114.209.209HTTP and/or

HTTPS traffic

Server Pool 1

Server Pool 2Rule 2/bar/

Rule 1/foo/



NSX-T Data Center TimelineKubernetes, OpenShift and PKS

2017 2018

September October November December January February March

NSX-T 2.1Support for PKS 0.8 and PKS 1.0

Support for K8s Ingress and Svc Type LB with Platform LB

Core value add:

• One of the only SDN solution in the market that includes LB with Ingress and Svc Type LB for K8s

• PKS / OPS MGR Integration

• Gives PKS support for Network Policy

NSX-T 2.0:Support for 'Do It Yourself' K8s & OpenShift

Core value add:

• Mapping of K8s Namespaces to Network Topology & source IP Addresses

• NAT & No-NAT modes per Namespace

• Network Policy (Firewall) across K8s and VM workloads

• Support for K8s Network Policy

• Logical Network Port per K8s workload (Pod) for visibility and troubleshootingVMworld 2018 Content: Not for publication or distribution


NSX-T TimelinePCF 2.0

2018

January February March April May June July

NSX-T 2.1Support for PCF 2.0 -> PAS

Core value add:

• Allows mapping of CF tenancy (Orgs) to Network Topology & source IP Addresses

• Network Policy (Firewall) support across PKS, PCF and VM workloads

• Only solution that allows for direct, no_NATcommunication from CF Apps to backend services

• Logical Network Port per CF workload (AI) for visibility and troubleshooting

NSX-T 2.2Operational Enhancement &Additional LB features

Core value add:

• Persistent SNAT IP for Kubernetes Services and CF Apps

• TLS/SSL Offload support for Kubernetes Ingress

• OpenShift 'router' support for HTPP and HTTPS (feature parity with K8s Ingress)

• URL rewrite support for K8s Ingress

• Various install & operational improvementsVMworld 2018 Content: Not for publication or distribution


NSX-T Data Center Values for Containers

Enterprise-class Networking

Advanced Security

Enhanced Operations

Full Network Visibility

Enterprise Support

Unified VM-to-Container Networking

Micro-Segmentation

N S X - T V a l u e s f o r C o n t a i n e r s

F e a t u r e sVMworld 2018 Content: Not for publication or distribution


Join the NSX VMUG Communityvmug.com/nsxConnect with your Peerscommunities.vmware.com

Embrace the NSX Mindsetnsxmindset.comFind NSX Resourcesvmware.com/go/networkingRead the Network Virtualization Blogblogs.vmware.com/networkvirtualization

Where to Get Started

Attend the Networking and Security SessionsShowcases, breakouts, quick talks & group discussions

Visit the VMware BoothProduct overviews, use-case demos

Visit Technical Partner BoothsIntegration demos – Infrastructure, security, operations, visibility, and more

Meet the ExpertsJoin our experts in an intimate roundtable discussion

Free Hands-on Labslabs.hol.vmware.com

Virtual Cloud Network Guided Demovcndemo.com

VMware Education – Training and Certificationvmware.com/go/nsxtraining

Free NSX Training on Courseravmware.com/go/coursera

Engage and Learn Experience

Try Take


http://vmug.com/nsx

https://communities.vmware.com/community/vmtn/nsx

http://www.vmwarensxmindset.com/

http://www.vmware.com/go/networking

http://blogs.vmware.com/networkvirtualization

http://labs.hol.vmware.com/

http://www.vcndemo.com/

http://www.vmware.com/go/nsxtraining

http://www.vmware.com/go/coursera

PLEASE FILL OUTYOUR SURVEY.Take a survey and enter a drawingfor a VMware company store gift card.

#vmworld #NET1677BU


THANK YOU!

#vmworld #NET1677BU


or distribution Networking with NSX-T for publication Kubernetes Container · 2018-09-05 ·...

Documents

Transcript of or distribution Networking with NSX-T for publication Kubernetes Container · 2018-09-05 ·...