Lessons Learned: Deploying

#vmworld

CNET1444BU

Lessons Learned: Deploying OpenShift with VMware SDDC

Vincent Han, VMware, Inc.Wayne Cheng, GovTech

#CNET1444BU

VMworld 2019 Content: Not for publication or distribution

©2019 VMware, Inc.

Disclaimer

This presentation may contain product features or functionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.

2

The information in this presentation is for informational purposes only and may not be incorporated into any contract. There is no commitment or obligation to deliver any items presented herein. VMworld 2019 Content: Not for publication or distribution

©2019 VMware, Inc.

Introduction

Staff Solution Specialist

Networking & Security

Vincent Han

3

Senior DevOps Engineer

Government Digital Services

Wayne Cheng


4©2019 VMware, Inc.

•Lessons learned during POC & Trial Implementation

•Knowledge to start your implementation

Why?


©2019 VMware, Inc.

Agenda

5

Customer Journey – GovTech Singapore

NSX Container Plugin (NCP) & Integration Overview

Lessons Learned and Next Steps

OpenShift on VMware SDDC

Q&A


©2019 VMware, Inc. 6

- Digitisation and Smart Nation Building

- Citizens’ interactions with Government

- Online government e-services

• https://www.tech.gov.sg/products-and-services/singapore-government-tech-stack/

- Platform as a Service – NECTAR

- API gateways – APEX

• https://www.tech.gov.sg/media/technews/getting-to-know-nectar-and-apex

Government Technology Agency of Singapore

Introduction to GovTech


https://www.tech.gov.sg/products-and-services/singapore-government-tech-stack/

https://www.tech.gov.sg/media/technews/getting-to-know-nectar-and-apex


Introduction to GovTech


8©2019 VMware, Inc.

•Failing is an option

•Strong technical competencies

•Find leverage

•Continuous learning, improvement & delivery



• Why we chose NSX-T as our SDN Solution

• Mixed workloads of Virtual Machines and Containers

• Adaptability of solution to integrate with Openshift Container Platform

• Provide native Load Balancing Service

• Granular control of network policies and visibility of the Containers’ Networks

• POC environment

• 1 Data Center, 3 Nested ESXi Hosts

• NSX-T 2.1, NCP 2.1, OSE 3.9

• 1 Master, 2 worker nodes

• Trial implementation environment

• 1 Data Center, 6 physical ESXi Hosts

• NSX-T 2.3, NCP 2.3, OCP 3.11

• 3 Masters, 2 Infra, 2 Worker Nodes

Requirements from GovTech and Why NSX-T



Journey so far…

▲

POC

OSE 3.9

NSX-T 2.1

NCP 2.1

Mar 2018 Apr 2018 May 2018 Dec 2018 Jan 2019 Feb 2019

▲

Trial Implementation

OCP 3.11

NSX-T 2.3

NCP 2.3



Kubernetes Node (VM)

Kubernetes Pod

Kubernetes 101

Stem B

Stem BStem B

Stem B Node

Node

Node

Kubernetes Master Node

API

Kubernetes Pod

App Container

RedisDB

Tools, Libs, SW

Pod 2

Pod 1

KDocker Engine

K

K

K

ESXi

App Container

RedisDB

Tools, Libs, SW

VM

VM

VM

VM

Stem BStem B

Stem BESXi

ESXi

vCenter

API VMVM

VM

CNI

NSX-T

NCP



NSX Container Plugin (NCP)

NCP infrastructure

NSX ManagerAPI Client

NSX Integration with CaaS / PaaS

NSX Manager

More…

OpenShift

PAS

Kubernetes

Hypervisor Bare-metal server

NSX Container Plugin Integrates NSX with Container Platforms

• Application deployment on container platforms leads to creation of networks, routers, firewalls, and load balancers

• No change to application platform user experience



Network Topology

KubeDNS

POD1

POD4

POD2

POD3

POD5

POD6

T0

Internet

kube-system PODs – Logical Switch - 10.12.0.0/24

Namespace ‘default’ PODs – Logical Switch - 10.12.1.0/24

Namespace ‘demo’ PODs – Logical Switch - 10.12.2.0/24

Master ‘VM’

etcdKubeDNS

APISrv

Node ‘VM’

Pod1

Pod2

Node ‘VM’

Pod3

Pod4

Node ‘VM’

Pod5

Pod6

T1

T1

T1

T1-Mgmt

Cluster Management Nodes – LS-VIFs (Non Routable)

Cluster Management Nodes – LS-Mgmt – 10.11.1.0/24

PhysicalRouter

Edges Uplink

ens192 ens192ens192ens224 ens224 ens224

NSX-TControllersvCenter

NSX-TManager

Secured RepoServer

Management Network – 192.168.110.0/24



NCP is NSX Container Plugin

• Translating Kubernetes resources to NSX-T objects

• When NCP starts, it will check both Kubernetes Resources and NSX-T objects and fill the gaps between them.

What is NCP

NCP Architecture

Openshift / Kubernetes Resources

Pod

Openshift Route / Ingress

Project / Namespace

Network Policy

NSX-T Objects

Service(Loadbalancer)

Container Interface

NSX-T L7 LB Rule and Pool

T1 Router, SNAT,,,etc

DFW

NSX-T L4 LB VS and Pool

NSX Container Plugin


©2019 VMware, Inc.

NCP ArchitectureNSX Container Plugin would work once all NCP components are properly installed

• NCPTranslating Kubernetes resources to NSX-T objects

• NSX-Node-AgentForward pod network informationPlumb pod interface to OpenvSwitch inside node vm

• NSX CNI pluginCNI interface between kubelet and nsx-node-agent

• NSX-Kube-ProxyTranslate Service(ClusterIP) of Kubernetes Resources into OpenvSwitch configuration

• OpenvSwitchProvide container networking and Service(Cluster IP)Isolate Pod traffic in VM

Host

K8s/OpenShift Node(VM)

kubelet

KubernetesControl

Plane

Pod Pod

NCP

LCP

OpenvSwitch

nsx-kube-proxy

vNIC

vmk50

CNI

nsx-node-agent


©2019 VMware, Inc.

OpenvSwitch and Container Network Security

• Distributed Firewall (DFW) at Host

• Micro-segmentation Pod to Pod traffic

Host

Node(VM)

br-int(ovs)

vNIC

VLAN 1 VLAN 2

CIF CIF

T1 DRns: B

Topology



Host

K8s Master(VM)

NCP Components

There are some components in NSX Container Plugin

• NCP

– Deployed as a Pod of Kubernetes.

– Only single instance is running on one of kubernetes node

• NSX Node Agent

– Deployed as a DaemonSet of Kubernetes

– Every Kubernetes nodes has NSX-Node-Agent

• OpenvSwitch

– Virtual switch installed inside Kuberntes nodes

– Every Kubernetes nodes has OpenvSwitch

• NSX CNI plugin

– CNI plugin used in NSX-T integration.

– Installed in every Kubernetes nodes

Kubernetes Resources

NSX-T Manager

NSX-Node-Agent(DaemonSet)

CNI OpenvSwitch

Host

K8s Node (VM)



K8s Node(VM)



NCP(Deployment)

CNI OpenvSwitch CNI OpenvSwitch



Installation

Failed multiple times due to various conditions

- Edge cannot be active-active

- Missing vmk 50

- NIC config (OOB mgmt, OCP mgmt, POD networking)

Documentation not readily available

Key use case not working

- Ver2.3 does not support custom certs, thus unable to test https route

Created a dependency for OCP to NSX-T

- Increased the complexity for day 2 operations, eg. Upgrading/Maintenance

Challenges



Trial Implementation Topology (Will not work)

KubeDNS

POD1

POD4

POD2

POD3

POD5

POD6

T0

Internet

kube-system PODs – Logical Switch

Namespace ‘default’ PODs – Logical Switch

Namespace ‘demo’ PODs – Logical Switch

Master ‘VM’

etcdKubeDNS

APISrv

Node ‘VM’

Pod1

Pod2

Node ‘VM’

Pod3

Pod4

Node ‘VM’

Pod5

Pod6

T1

T1

T1


PhysicalRouter

Edges Uplink


NSX-TManager

Secured RepoServer




Trial Implementation Topology Final Working

KubeDNS

POD1

POD4

POD2

POD3

POD5

POD6

NSX-TControllers

T0

Internet




Master ‘VM’

etcdKubeDNS

APISrv

Node ‘VM’

Pod1

Pod2

Node ‘VM’

Pod3

Pod4

Node ‘VM’

Pod5

Pod6

T1

T1

T1

T1-Mgmt

PhysicalRouter

vCenterNSX-T

Manager

Edges Uplink

Secured RepoServer






• Check supportability matrix before installation

• https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.4/rn/NSX-Container-Plugin-Release-Notes.html

• Ensure Reserve IP Range not used - Default Service IP Range - 172.30.0.0/16

• Size of NSX-T Edge VMs

• https://communities.vmware.com/docs/DOC-40435

• T0 – Active/Standby

• NSX-T Principal Identity

Tips before Installation


https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.4/rn/NSX-Container-Plugin-Release-Notes.html

https://communities.vmware.com/docs/DOC-40435


1. Install NSX-T and setup Infra

2. Place NCP CNI Package and OVS package on Secured Repo Server

3. NSX-T resource setup

• Create IP block used for pod network

• Create IP block used for No SNAT

• Create IP pool used for LB VIP

4. Put tags on vNIC

5. Upload NCP Docker image on every node

6. Amend Ansible Host file and deploy OpenShift cluster

7. Deploy NCP [Part of deploy-cluster.yml in OpenShift 3.11]

8. Deploy NSX node agent [Part of deploy-cluster.yml in OpenShift 3.11]

https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.4/nsxt_24_ncp_openshift.pdf

NCP Installation Overview


https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.4/nsxt_24_ncp_openshift.pdf


NCP is available in my.vmware.com

The package has

• NCP deployment yaml file

• NSX-Node-Agent yaml file

• OpenvSwitch packages

• NCP docker image.

NCP installationNCP packages



Internal Structure

NSX Container Plugin

CNI and NSX Node Agent yaml per OS

NCP Docker Images

Openvswitch packages

Deployment is preferred. RC is legacy.

Required for OCP 3.11





https://github.com/vincenthanjs/openshift-ansible-hosts/blob/master/hosts

OpenShift Ansible hosts file

openshift_master_default_subdomain=ocpapps.acepod.comopenshift_use_nsx=trueos_sdn_network_plugin_name=cniopenshift_use_openshift_sdn=falseopenshift_node_sdn_mtu=1500

# NSX specific configurationnsx_openshift_cluster_name='ocp-cl1'nsx_api_managers='192.168.110.26'nsx_api_user='admin'nsx_api_password='VMware1!'nsx_tier0_router='JUR01-T0'nsx_overlay_transport_zone='TZ-Overlay'nsx_container_ip_block=‘IP-Block-OCP-Container'nsx_no_snat_ip_block=‘IP-Block-OCP-NO-SNAT'nsx_external_ip_pool=‘IP-Pool-OCP-External'nsx_top_fw_section='openshift-top'nsx_bottom_fw_section='openshift-bottom'nsx_ovs_uplink_port='ens224'nsx_cni_url='http://192.168.110.12/nsx-cni-2.3.2.11695762-1.x86_64.rpm'nsx_ovs_url='http://192.168.110.12/openvswitch-2.9.1.9968033.rhel75-1.x86_64.rpm'nsx_kmod_ovs_url='http://192.168.110.12/kmod-openvswitch-2.9.1.9968033.rhel75-1.el7.x86_64.rpm’

Need a secured repo server to host the files.





NSX-T Configuration

openshift_master_default_subdomain=ocpapps.acepod.comopenshift_use_nsx=trueos_sdn_network_plugin_name=cniopenshift_use_openshift_sdn=falseopenshift_node_sdn_mtu=1500

# NSX specific configurationnsx_openshift_cluster_name='ocp-cl1'nsx_api_managers='192.168.110.26'nsx_api_user='admin'nsx_api_password='VMware1!'nsx_tier0_router='JUR01-T0'nsx_overlay_transport_zone='TZ-Overlay'nsx_container_ip_block=‘IP-Block-OCP-Container'nsx_no_snat_ip_block=‘IP-Block-OCP-NO-SNAT'nsx_external_ip_pool=‘IP-Pool-OCP-External'nsx_top_fw_section='openshift-top'nsx_bottom_fw_section='openshift-bottom'nsx_ovs_uplink_port='ens224'nsx_cni_url='http://192.168.110.12/nsx-cni-2.3.2.11695762-1.x86_64.rpm'nsx_ovs_url='http://192.168.110.12/openvswitch-2.9.1.9968033.rhel75-1.x86_64.rpm'nsx_kmod_ovs_url='http://192.168.110.12/kmod-openvswitch-2.9.1.9968033.rhel75-1.el7.x86_64.rpm’

NSX-T Manager Configurations

OCP Topology

POD1

POD4

T0

Namespace ‘foo’ PODs – Logical Switch – 10.12.5.0/24

Master ‘VM’

etcdKubeDNS

APISrv

Node ‘VM’

Pod1

Node ‘VM’

Pod2

T1

T1-Mgmt



External LB IP Pool – 10.21.0.0/24

SNAT IP for Namespace foo - 10.21.0.2

T1-LB

ens192 ens224 ens192 ens224VMworld 2019 Content: Not for publication or distribution



VIFs Tagging

OCP Topology

POD1

POD4

T0

Namespace ‘foo’ PODs – Logical Switch – 10.12.5.0/24

Master ‘VM’

etcdKubeDNS

APISrv

Node ‘VM’

Pod1

Node ‘VM’

Pod2

T1

T1-Mgmt



External LB IP Pool – 10.21.0.0/24

SNAT IP for Namespace foo - 10.21.0.2

T1-LB

ens192 ens224 ens192 ens224



• Blogs

• https://blogs.vmware.com/networkvirtualization/2019/02/nsx-t-integration-with-openshift.html/

• http://blog.acepod.com/how-to-install-openshift-container-platform-ocp-with-nsx-t-ncp/

• Installation Videos

• NSX-T Openshift 3.11 Installation/Integration demo - https://youtu.be/uEQ5UAgh770

• How to install Openshift Container Platform 3.11 Enterprise with VMware NSX-T (Native Integration) - https://youtu.be/5ZlggXKXwL8

• Sample Openshift Ansible host file

• https://github.com/vincenthanjs/openshift-ansible-hosts/blob/master/hosts

Resources for your reference


https://blogs.vmware.com/networkvirtualization/2019/02/nsx-t-integration-with-openshift.html/

http://blog.acepod.com/how-to-install-openshift-container-platform-ocp-with-nsx-t-ncp/

https://youtu.be/uEQ5UAgh770

https://youtu.be/5ZlggXKXwL8



Reference Topology

KubeDNS

POD1

POD4

POD2

POD3

POD5

POD6

T0

Internet




Master ‘VM’

etcdKubeDNS

APISrv

Node ‘VM’

Pod1

Pod2

Node ‘VM’

Pod3

Pod4

Node ‘VM’

Pod5

Pod6

T1

T1

T1

T1-Mgmt



PhysicalRouter

Edges Uplink

ens192 ens192ens192ens224 ens224 ens224


NSX-TManager

Secured RepoServer




VMs Placement

Compute ClusterManagement/Edge Cluster

vCenter

NSX-TControllers

NSX-TManager

NSX-TEdge VMs

VM

VM

VM

VM

VM

VM

OCP Master Nodes

OCP Worker Nodes

ESXi ESXi



POC Topology (Nested ESXi for POC)

KubeDNS

POD1

POD4

POD2

POD3

POD5

POD6

T0




Master ‘VM’

etcdKubeDNS

APISrv

Node ‘VM’

Pod1

Pod2

Node ‘VM’

Pod3

Pod4

Node ‘VM’

Pod5

Pod6

T1

T1

T1

T1-Mgmt



Internet

PhysicalRouter

Edges UplinkRouter VM


NSX-TManager

Secured RepoServer




POC VMs Placement (Nested ESXi for POC)

Compute ClusterManagement/Edge Cluster

vCenter

NSX-TControllers

NSX-TManager

NSX-TEdge VMs

VM

VM

VM

VM

VM

VM

OCP Master Nodes (Nested)

OCP Worker Nodes (Nested)

ESXi ESXi

nested-ESXi (VM) nested-ESXi (VM)



Lessons learned and key considerations



• Upgrade NCP from 2.3.x to 2.4.x

• Test vRealize Network Insight vRNI 4.2• Container Network Flows

• Supporting Kubernetes & OpenShift

• Detailed Network Policy

Next Steps



• VMware and Red Hat announce on May 9, 2019 to collaborate to better integrate OpenShift Container Platform and VMware Software Defined Data Center.

• Simplify networking and network-based security with the NSX Container Plug-in (NCP)

• vSphere Cloud Provider and its corresponding volume plugin –vSAN or any vSphere datastore

OpenShift Container Platform on VMware SDDC

35



Other VMworld Sessions - Red Hat and VMware



Deep Dive on NSX-T with Kubernetes Networking



NSX-T Data CenterCloud-Native Network Services Platform for Cloud-Native Apps

On-Premises – vSphere, Bare-metal and KVM

Business App 2 / LOB 2

CF K8s

Business App 1 / LOB 1

CF K8s

NSX Platform

• Common networking model

• Agility and lower costs with scale out load balancing and firewall

• Monitoring, troubleshooting, and audit controls



• NSX Container Plugin provides tremendous value in networking & security for OpenShift/Kubernetes

• Start your own journey

• Share your experiences with us• Vincent Han - @vincenthan at Twitter or LinkedIn

• Wayne Cheng @LinkedIn

Key Takeaways


Lessons Learned: Deploying

Documents

Transcript of Lessons Learned: Deploying