NET1350BE Deploying NSX on a Cisco Infrastructure or ... · PDF fileDeploying NSX on a Cisco...

Jacob Rapp – [email protected] Jablonski – [email protected]

NET1350BE

#VMworld #NET1350BE

Deploying NSX on a Cisco Infrastructure

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

2#NET1350BE CONFIDENTIAL



bution

Agenda

1 NSX Anywhere

2 NSX Design

3 Nexus Switching Fabric

4 UCS Connectivity

5 NSX on ACI

6 Summary & Question

#NET1350BE CONFIDENTIAL 3



bution

Agenda

1 NSX Anywhere

2 NSX Design


4 UCS Connectivity

5 NSX on ACI





bution

Apps Driving Infrastructure Requirements

Works across hypervisors,

application frameworks, clouds

Infrastructure

independent, with

standards-based

interoperability where

necessary

Security wrapped around the

VM, container & microservice

People, Process and Tooling Model#NET1350BE CONFIDENTIAL 5



bution

Infrastructure Independent

Consistent policy

Co

ntr

olle

d

Co

mm

un

ica

tio

n

Stateful DFW

3rd Party Services

Stateful DFW

Co

ntr

olle

d

Co

mm

un

ica

tio

n

Stateful DFW

3rd Party Services

Stateful DFW

Co

ntr

olle

d

Co

mm

un

ica

tio

n

Stateful DFW

3rd Party Services

Stateful DFW

Stateful DFW

3rd Party Services

Micro-Segment

Co

ntr

olle

d

Co

mm

un

ica

tio

n

Stateful DFW

3rd Party Services

Stateful DFW

Co

ntr

olle

d

Co

mm

un

ica

tio

n

Stateful DFW

3rd Party Services

Stateful DFW

Co

ntr

olle

d

Co

mm

un

ica

tio

n

Stateful DFW

3rd Party Services

Stateful DFW

Stateful DFW

3rd Party Services

Stateful DFW

Micro-Segment

Universal Distributed Logical Router

Any IP network

Up to 150ms

Public clouds

Any TransportACI Fabric

Single/Multi PodOr

Any L2/L3 Fabric

Any L2/L3Fabric

Stateful DFW




bution

Security Everywhere

ContextControl Points

Data Center

Cloud

Campus/Branch

Mobile

Endpoint

Telemetry:

Application

Network

Intelligence

Policy

Complete framework

spanning use cases

and service stack




bution

Multi-Hypervisor, Cloud and Frameworks

Automation

IT at the Speed of Business

Security

Inherently Secure Infrastructure

Application Continuity

Data Center Anywhere

On-Premise Data Center

New app frameworks

Mobile Devices(Airwatch)

Virtual Desktop(VDI)

Branch offices

Public clouds

vCloud AirNetwork




bution

People, Process and Tooling Model

Ending State

Beginning State

Blended,

cross-functional

Siloed,

specialized

People

Speed, agility,

standardization

Slow, error prone,

inconsistent

Processes

Converged,

correlated

Domain-specific,

hardware focused

Tooling

Reseach paper detailing operational changes: https://tinyurl.com/y8dme6gx#NET1350BE CONFIDENTIAL 9



bution

Agenda

1 NSX Anywhere

2 NSX Design


4 UCS Connectivity

5 NSX on ACI





bution

Software Defined Data Center Delivers Freedom

Stateful DFW

3rd Party Services

Stateful DFW

Micro-Segment

Stateful DFW

3rd Party Services

Stateful DFW

Micro-Segment

Stateful DFW

3rd Party Services

Stateful DFW

Micro-Segment

Stateful DFW

3rd Party Services

Stateful DFW

Micro-Segment

Stateful DFW

3rd Party Services

Stateful DFW

Micro-Segment

Stateful DFW

3rd Party Services

Stateful DFW

Micro-Segment

Stateful DFW

3rd Party Services

Stateful DFW

Micro-Segment

Stateful DFW

3rd Party Services

Stateful DFW

Micro-Segment

Difficult operational model

Complex protocol mix

Hardware ASIC dependent

Finite service offering

Network hardwarecentric application and

service deployment

Decouple applications

from hardware infrastructure:

Simple, Scalable Service model




bution

NSX vSphere Cluster Design

Physical Network

Compute Clusters Edge Cluster(s) Management Cluster

Application

Transport Subnet A

192.168.150.0/24

Transport Subnet B

192.168.250.0/24

NSX EdgeNSX Controller

ClusterNSX

Manager

VM1 VM2

VM3VM5

VM5

Management

(VLAN)

Database

Transit VXLAN

Web

Tiered

Logical Switches

VM6

Transport Zone (VLAN)

ControlVM




bution

VMkernel Networking

vSphere Host (ESXi)

13

Layer 2 or Layer 3 Uplinks

VLAN Trunk (802.1Q)

VLAN 66

Mgmt

10.66.1.2/26

DGW: 10.66.1.1

VLAN 77

vMotion

10.77.1.2/26

DGW: 10.77.1.1

VLAN 88

VXLAN

10.88.1.2/26

DGW: 10.88.1.1

VLAN 99

Storage

10.99.1.2/26

DGW: 10.99.1.1

SVI 66: 10.66.1.1/26

SVI 77: 10.77.1.1/26

SVI 88: 10.88.1.1/26

SVI 99: 10.99.1.1/26

Sp

an

of V

LA

NsS

pa

n o

f V

LA

Ns

IP Stacks– Default– vMotion– VXLAN




bution

NSX VXLAN Data and Control Plane

• NSX VXLAN Control Plane

– Unicast mode or Hybrid Mode

• Unicast preferred with ACI infrastructure

• NSX Data Plane

– VMKernel VTEP encaps/decaps

– Transport Zone

VM Sends a standard L2 Frame

Source Hypervisor (VTEP)encapsulates VXLAN, UDP &

IP Headers

Destination Hypervisor (VTEP) decapsulate

headers

Physical Network forwards frame as standard IP frame

Original L2 Frame delivered

to VM

VXLAN

VTEP

VXLAN

VTEP

IP FrameL2 Frame L2 Frame

1

2 43 5

L2 IP UDP VXLAN PayloadL2

1

2

3

4

3

UDP VXLAN PayloadL2

5




bution

NSX VXLAN Infrastructure

• Transport Zone

– VXLAN prepared ESXi clusters

– VTEP(s)

– 1 prepared VDS per cluster enabled for VXLAN

– Logical Switch as dvPortgroup

– Support for VMware vDS managed by vSphere

vSphere

Host

VXLAN Transport

Network

10.20.10.1010.20.10.11

VTEP1 VTEP2

VM

VXLAN 5002 MAC2

vSphere

Host10.20.10.12 10.20.10.13

VM

MAC4

VM

MAC1

VM

MAC3

VTEP3 VTEP4

Compute VDS Edge VDS

VXLAN prepared

Hosts




bution

VDS Uplink Design

• NSX host preparation

– Creates VXLAN dvUplink

• Consistent for all hosts using that VDS

• Consistent teaming policy

– Recommended teaming mode

• Route Based on Originating Port

• LACP is not possible from UCS blade

Teaming

Mode

NSX

Support

Multi-

VTEP

Uplink

2 x 10G/40G

Nexus/ACI

Port

Configuration

Route based on

Originating Port✓ ✓ All Active Standard

Route based on

Source MAC

hash

✓ ✓ All Active Standard

LACP ✓ ×Flow based

All active

vPC Port-Channel -

LACP

Route based on

IP Hash (Static

EtherChannnel)

✓×

Flow based

All activevPC Port-Channel

LACP mode OFF

Explicit Failover

Order✓ ×

Single link

activeStandard

Route based on

Physical NIC

Load (LBT)

× × × Standard




bution

Connect Your Workloads to a Physical Network

• Route and switch where you can. • Bridge if you must.

Physical

Network

Compute/Edge

Clusters

Ea

ch

brid

ge

d s

egm

en

t is

a s

ep

ara

te s

ub

ne

t

VM1 VM2

VM3VM5

VM5

Application

Database

Web

VLAN

VXLAN

VL

AN

su

bn

ets

T

ran

sit s

ub

ne

t

VX

LA

N s

ub

ne

ts

VM1 VM2

VM3VM5

VM5




bution

Agenda

1 NSX Anywhere

2 NSX Design


4 UCS Connectivity

5 NSX on ACI





bution

NSX Infrastructure Requirements

NSX is AGNOSTIC to underlay network topology

L2 or L3 switched infrastructure

Only Two Requirements

2) IP Connectivity

1) MTU of 1600




bution

Jumbo MTU – Nexus 5000 and 6000

• VDS Max MTU is 9000 Byte

• Nexus 5xxx, 56xx and 6xxx requires

– MTU to be changed with Policy-Map

– L3 requires per interface MTU change

• All links belonging to fabric must be enabled with Jumbo MTU

All L2 interfaces Layer 3 Interface

Only global configurations

Create policy-map:

policy-map type network-qos jumbo

class type network-qos class-default

mtu 9216

Apply policy-map:

system qos

service-policy type network-qos jumbo

interface Vlan151 SVI Interface

no ip redirects

ip address 10.114.221.34/27

hsrp 1

ip 10.114.221.33

description VXLAN Transport Zone

no shutdown

mtu 9216

interface Ethernet2/12 Layer 3 Interface

description L3 Link to Spine

no switchport

speed 40000

duplex full

mtu 9216

ip address 10.114.211.117/31

no shutdown




bution

Jumbo MTU – Nexus 7000 and 9000

• VDS Max MTU is 9000 Byte

• Nexus 7xxx, 9xxx Series

– L2 only requires global configuration

– L3 requires per interface MTU change

• All links belonging to fabric must be enabled with Jumbo MTU

Layer 2 Interface Layer 3 Interface

system jumbomtu 9216 Global configurations

interface Ethernet1/9

description to esx-vmnic3-VMK

switchport mode trunk

switchport trunk allowed vlan 22-25

spanning-tree port type edge trunk

mtu 9216 Layer 2 MTU

channel-group 9 mode active

interface Vlan151 SVI Interface

no ip redirects

ip address 10.114.221.34/27

hsrp 1

ip 10.114.221.33

description VXLAN Transport Zone

no shutdown

mtu 9216

interface Ethernet2/12 Layer 3 Interface

description L3 Link to Spine

no switchport

speed 40000

duplex full

mtu 9216

ip address 10.114.211.117/31

no shutdown




bution

Cisco DC Topology – L2 Pod – NSX is Agnostic

VLANs & IP Subnet Defined at 95xx for

POD A

SVI Interface VLAN ID IP Subnet

Management 100 10.100.A.x/24

vMotion 101 10.101.A.x/24

Storage 102 10.102.A.x/24

VXLAN 103 10.103.A.x/24

VLANs & IP Subnet Defined at 95xx for

POD B


Management 200 10.200.B.x/24

vMotion 201 10.201.B.x/24

Storage 202 10.202.B.x/24

VXLAN 103 10.103.B.x/24

VXLAN VLAN ID 103 - Transport Zone Scope (extends across ALL PODs/clusters)

Compute

Cluster A

Compute

Cluster B

VLAN ID 100, 101 & 102 Scope VLAN ID 200, 201 and 203 Scope

POD A

L3

L2

UCS B-Series

POD B

UCS B-Series

L3 Core

L3

L2

95xx95xx

95xx 95xx 95xx 95xx

93xx 93xx 93xx 93xx 93xx 93xx 93xx 93xx

Pod

components

can be any

mix of 9k /

7k/ 6k / 5k /

2k




bution

Cisco DC Topologies – L3 Design – NSX is Agnostic

VLANs & IP Subnet Defined

at each ToR


Management 100 10.100.R_ID.x/24

vMotion 101 10.101.R_ID.x/24

Storage 102 10.102.R_ID.x/24

VXLAN 103 10.103.R_ID.x/24

VXLAN VLAN ID 103 - Transport Zone Scope (extends across ALL PODs/clusters)

Compute

Cluster A

Compute

Cluster B

VLAN ID 100, 101 & 102 Scope

POD A

UCS B-Series

POD B

UCS B-Series

L3 Core

95xx95xx

95xx 95xx 95xx 95xx

93xx 93xx 93xx 93xx 93xx 93xx 93xx 93xxL3

L2

L3

L2

VLAN ID 100, 101 & 102 Scope




bution

Cisco DC Topologies – VXLAN

VLAN ID 100, 101 and 102 Scope – VXLAN VLAN ID 103 - Transport Zone Scope (extends across ALL PODs/clusters)

Compute

Cluster A

Compute

Cluster B

UCS B-Series UCS B-Series

Spine

Leaf

UCS B-Series UCS B-Series

Border Leaf

Mgt / Edge

Cluster

VLANs & IP Subnet Defined at

each ToR


Management 100 10.100.100.x/24

vMotion 101 10.101.101.x/24

Storage 102 10.102.102.x/24

VXLAN 103 10.103.103.x/24

L3 Spine DC Core

Internet/DMZ

56xx 56xx 93xx 93xx 93xx 93xx

95xx 95xx 95xx 95xx

Spine - Leaf

can be:

9xxx, 7xxx,

6xxx , 56xx

60xx 60xx

Cisco’s Prime or NFM may also

provide underlay and VXLAN

management.

(NFM supports only 9ks)




bution

Agenda

1 NSX Anywhere

2 NSX Design


4 UCS Connectivity

5 NSX on ACI





bution

NSX Connectivity Ideals

vSphere Host and UCS Interconnectivity

• UCS Fabric Interconnects

– End-Host mode

– vPCs to Nexus switching

• vSphere Compute connectivity

– UCS vnics shared or dedicated

– vSphere dvUplinks equals number of vmnics

– VTEPs with Src ID teaming

• vSphere Edge connectivity

– Preferably UCS C-Series

– Separate connection, bypass FIs

95xx

UCS

Edge Leaf

L3

L2

95xx

93xx

Edge Cluster

UCS C-Series

DC Core

Internet/DMZ

DC Fabric

93xx

VPN

VPN

VPN

VPN




bution

NSX Edge Routing for UCS C Series & Intel NICs

Recommended design for Edge routing:

• Two Uplinks per ESG

• Per ESG, establish adjacency to each ToR (redundancy)

– VLAN backed networks routing neighbors on the ToRs A/B

– Map each VLAN of the dvPortgoup to each dvUplink

– VLANs used for dynamic routing are local to each router

– Use Source ID

• eBGP is used between NSX ESG and routers A/B

• Equally applicable to OSPF

• Default route must follow the uplink status

– Loss of Uplinks will withdraw routes on that fabric and upstream link

VLAN 10 SVI VLAN 20 SVI

vNIC1

Uplink A

VLAN 10

vNIC2

Uplink B

VLAN 20eBGP

Peering

Default route

advertised downstream

9K-B9K-A

95xx 95xx

vDS-Edge

Pair of NSX

Edges per

ESXi host




bution

NSX Edge Routing for UCS B Series

Recommended design for Edge routing

• Dedicated UCS vnics pinned for routing

• Port-Channel connection between FI and ToRs

– Redundancy and scaling bandwidth

• Per ESG, establish an adjacency per ToR (Redundancy)

– Each ESG, bound to one active dvUplink

• eBGP is used between NSX ESG and N9Ks

• Equally applicable to OSPF

• Default route must follow the uplink status

– Loss of Uplinks will withdraw routes on that fabric and upstream link

Dedicated non-vPC

pinned

1 vNIC maps to

Fabric A

VLAN 10

1 vNIC maps to

Fabric B

VLAN 20

UCS FI-A UCS FI-B

Pair of NSX

Edges per

ESXi host

VLAN 10 SVI VLAN 20 SVI9K-B9K-A

vDS-Edge

Default route

advertised downstream

95xx 95xx

eBGP

Peering




bution

VDS Design, Uplink & Traffic Mapping

Recommended UCS B-Series setup

VMNIC 0 VMNIC 1 VMNIC 2 VMNIC 3

vNIC 3vNIC 1

2204 FEX2204 FEX

6248 (A) 6248 (B)

Teaming

Mode

VMkernel

VXLAN

VTEP – 1

Traffic

Type

VMkernel

VXLAN

VTEP - 2

VMkernel

vMotion

VMkernel

MgmtVMkernel

IP Storage

LBT SRC_ID SRC_ID Explicit

FailoverLBT

Nexus 93XXNX-OS Mode

vNIC 4vNIC 2

VDS – 2 Routing

Routing

VLAN PG 20

Bridging

PGRouting

VLAN PG 10

VDS - 1 MGMT, vMotion, NFS, VXLAN & Bridging

93xx 93xx

SRC_ID SRC_ID SRC_ID

UCSB-SeriesBlade




bution

Agenda

1 NSX Anywhere

2 NSX Design


4 UCS Connectivity

5 NSX on ACI





bution

NSX Network Virtualization with ACI underlay

vCenterNSX

NSX Network Virtualization provides

• Cloud automation

• Integrated NSX service deployment

• Embedded security deployment

• Provides P 2 V integrated services and security

• Inherent Services: LB, DHCP, NAT, VPN,

and 3rd party service insertion

Customer Benefits

• No dependency on infrastructure for service mobility

• Choice of underlay network

• Opportunistic leveraging of fabric

• E.g -> ACI and network infrastructure isolation

Cloud Management Platform

vRealize, OpenStack, Custom




bution

NSX Over ACI Recommendations

Fabric

Infrastructure

ACI

Fabric Ideals

Layer 2 Fabric:

Single Tenant

Fewer Contract Needs

Map Static vSphere EPs

Map NSX Edge to ACI Border

Minimum Requirements:

1 Physical Domain

1 External Routing Domain

2 VLAN Pools (Int & Ext)

1 AEP (Leaf & Switch Policies,

Int & Int Sel Policies, etc..)

NSX over ACI

Tenant

Tenant:

Separate tenant (not common)

1 Application (Network) Profile

4 EPGs (base epgs)

4 Bridge Domains, 1 VRF

2 L3Outs; North and South




bution

NSX Over ACI: ACI Infrastructure

ACI Infrastructure

• Supports attachment of hosts

– Define phy domain of host attachment

– VLANs, switch interfaces, and policies in use

– Domains, Physical and External

• Create Application Profile

– Defines EPGs

– Networks

• Private Networks

• Bridge Domains

• External L2 and L3 connectivity

NSX Overlay

Compute

VMKernel

ACI Contracts

ACI EPGs

Layer 2

ACI Fabric

Compute VDS Edge VDS

VM1 VM3VM5

VM2 VM4VM6

Web LS

APP LS

DB LS

Peer VLANs

MGMT vMotion Storage Transport




bution

ACI Infrastructure Screen Shot

4 EPGs

Contracts unnecessary for Intra EPG communication

Specify Physical domain and define Static Ports along with a VLAN

encapsulation




bution

NSX Edge Mapping to ACI Border Leaf

• Compute Clusters

– Compute workloads

– Services (Tenant LB, NAT, etc)

– Distributed routing, switching, DFW

• Edge Clusters

– ECMP routing with ACI border leafs

– ESGs single active uplink per dvUplink

Border Leafs

ECMP Edges

EPG/VLAN 20

L3Out…

EPG/VLAN 10

L3Out

L3

L2

DC CoreACI Spine

North bound L3Outs

RoutingAdjacency

VM1 VM3 VM5

Compute Edge

ControlVM

Vmkernel

VXLAN Overlay




bution

NSX Edge BGP AS Configuration (Prod)

• BGP Connectivity

– DC Core to ACI Border Leafs

• eBGP connection

– DC Core: AS 1

– ACI Leafs: AS 65100

– ACI Border Leafs to ESGs

• eBGP connection

– ACI Leafs: AS 65100

– ESGs: 65014

– ESGs to DLR CVM

• eBGP connection

– ESGs: 65014

– DLR Control VM: 65013

– ACI Spines – MPBGP

• AS 65001

• This AS must NOT be used by ACI for L3Out Neighboring

E1

Border Leafs

E3

EPG/VLAN 1210

L3Out

EPG/VLAN 1209

L3Out

L3

L2

DC Core

ACI Spine

L3Outs

North

EdgeCluster

ControlVM

Vmkernel

VXLAN Overlay

E2 E4

BGP65014

BGP65001

BGP1

BGP65013

BGP65100




bution

ACI Tenant Networking

Networking• Bridge domains• VRFs or Private Networks• vzAny contract for EPG

collection for VRF • Allow all L3Out and IP

Mgmt traffic to communicate externally

External Routed Networks• NSX to ACI border leafs• ACI border leafs to Nexus DC

cloud (not pictured)• Default Route Leak Policy

• Allow Default route




bution

ACI L3Outs – NSX Edges

L3 External Routing• Per ACI Leaf config• BG Peer defined• Utilize defined SVI per

leaf, per VLAN

SVI’s • IP address per leaf node• Define switch node, ports

and transit vlan




bution

ACI BGP Peer Connectivity

BGP Peer Connectivity Profile

Define BGP peer, BGP Prefix policy for

prefix distribution, and relative BGP

neighbor attributes




bution

NSX Edge Mapping to ACI Border Leaf

• L3Out Edge Connectivity

– Transit VLANs

• 1209: 10.114.219.208/29

– Edges: E1:210, E3:211

– ACI SVI: .209

• 1210: 10.114.219.216/29

– Edges: E2: .218, E3: .220

– ACI SVI: .217

E1

Border Leafs

ECMPE3

10.114.219.216/29

VLAN 1210

10.114.219.208/29

VLAN 1209

L3

L2

DC CoreACI Spine

L3Outs

North

Compute

EdgeCluster

ControlVM

Vmkernel

VXLAN Overlay

E2 E4

Transit VXLAN

Transit VXLAN 10.114.219.184/29

DLR Forwarding IP: .185

DLR Protocol IP: .186

E1 – E4: (.187 – 190)




bution

Multi Site Heterogenous DR

L2/L3 DCI

Any L2/L3 DCI

Policy Everywhere- Stateful Firewall- Network Introspection- Identity Firewall- Decoupled - Less replacement

cycles

Universal

Logical

Switches




bution

Agenda

1 NSX Anywhere

2 NSX Design


4 UCS Connectivity

5 NSX on ACI





bution

NSX Design Guides

• Reference Design: Deploying NSX with Cisco UCS and Nexus 9000 Infrastructurehttps://communities.vmware.com/docs/DOC-29373

• Design Guide for Vmware NSX running with a Cisco ACI Underlay Fabrichttps://communities.vmware.com/docs/DOC-30849

• NSX-V Multi-Site Options and Cross-VC NSX Design Guidehttps://communities.vmware.com/docs/DOC-32552

• VMware® NSX for vSphere Network Virtualization Design Guide version 3.0https://communities.vmware.com/docs/DOC-27683




bution

https://communities.vmware.com/docs/DOC-29373






bution

NET1350BE Deploying NSX on a Cisco Infrastructure or ... · PDF fileDeploying NSX on a Cisco...

Documents

Transcript of NET1350BE Deploying NSX on a Cisco Infrastructure or ... · PDF fileDeploying NSX on a Cisco...