Best Practices for Deploying VXLAN with Nexus 1000V and vCloud Director

Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 1

Best Practice for Deploying VXLAN with Cisco Nexus 1000V and VMware vCloud DirectorHan YangProduct Manager, Data Center Group

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2

Virtual Appliance Nexus 1010

vWAAS VSG VSM

NAM

NAM

VSG

VSG

Primary

Secondary

VSM

VSM

Cisco Nexus 1000 Portfolio

2

L3

Co

nn

ect

ivity

VSM: Virtual Supervisor Module

VEM: Virtual Ethernet Module

vPath: Virtual Service Data-path

VXLAN: Scalable Segmentation

VSG: Virtual Security Gateway

vWAAS: Virtual WAAS

ASA 1000V: Tenant-edge security

Virtual Service BladesVirtual Supervisor Module (VSM)

Network Analysis Module (NAM)

Virtual Security Gateway (VSG)

Data Center Network Manager (DCNM)

VEM-2

vPath

Win Server 2012

VXLAN

VEM-1

vPath

VMware ESX

VXLAN

ASA 1000V

VXLAN• 16M address space for LAN

segments

• Network Virtualization (Mac-over-UDP)

vPath• Service Binding (Traffic Steering)

• Fast-Path Offload

VEM-3

vPath

Open Source Hyp

VXLAN


Cisco Virtual Networking and Security SolutionNexus 1000V, CSR 1000V, ASA 1000V, VSG, and vWAAS Deployment

Nexus 1000V

• Distributed switch

• NX-OS consistency

VSG

• VM-level controls

• Zone-based FW

ASA 1000V

• Edge firewall, VPN

• Protocol Inspection

vWAAS

• WAN optimization

• Application traffic

Multi-Hypervisor

WAN Router

SwitchesServers

Tenant A

ASA 1000V

Zone BZone A

Nexus 1000VvPath

Physical Infrastructure

Virtualized/CloudData Center

vWAAS

VSG

VXLAN

CSR 1000V(Cloud Router)

• WAN L3 gateway

• Routing and VPN


Why VXLAN?


Virtual Workload on Physical Data Center

VM VM VM VM VM VM VM VM VM

Layer 2

Layer 2

VM VM

Elastic Virtual Workload VM VM

On Physical Server & Network Infrastructure

How to Optimally Leverage Physical Infrastructure?

How to Optimally Leverage Physical Infrastructure?

New Workload Exceeding Capacity

Mobility Across Layer 3?Mobility Across Layer 3?

Layer 3


Virtual Overlay Network with VXLAN

VM VM

Virtual Overlay Nework Crossing Layer 3

Utilize All Links in Port Channel w/ UDP

Add More Pods to Scale

VM VM VM VMVM


Virtual Overlay Network

VMData Center

Network

WAN

PhysicalFirewall

Bare Metal Servers

Router

Gateway

Gateway

Gateway

Overlay

• Overlay: Instant provisioning• Overlay needs gateway to access

physical network• Physical network to support overlay

traffic pattern


Virtual Extensible Local Area Network (VXLAN)• Ethernet in IP overlay network

Entire L2 frame encapsulated in UDP

50 bytes of overhead

• Include 24 bit VXLAN Identifier16 M logical networks

Mapped into local bridge domains

• VXLAN can cross Layer 3

• Tunnel between VEMsVMs do NOT see VXLAN ID

• IP multicast used for L2 broadcast/multicast, unknown unicast

• Technology submitted to IETF for standardization

With VMware, Citrix, Red Hat, Broadcom, Arista, and Others

Outer MACDA

Outer MACSA

Outer 802.1Q

Outer IP DA

Outer IP SA

Outer UDP

VXLAN ID (24 bits)

Inner MAC DA

InnerMAC

SA

Optional Inner 802.1Q

Original Ethernet Payload

CRC

VXLAN Encapsulation Original Ethernet Frame


VXLAN Forwarding Basics• Forwarding mechanisms similar to Layer

2 bridge: Flood & LearnVEM learns VM’s Source (MAC, Host VXLAN IP) tuple

• Broadcast, Multicast, and Unknown Unicast Traffic

VM broadcast & unknown unicast traffic are sent as multicast

• Unicast TrafficUnicast packets are encapsulated and sent directly (not via multicast) to destination host VXLAN IP (Destination VEM)

VM VMVM VM

VEM 1 VEM 2


WebVM

WebVM

DBVM

DBVM

Join Multicast Group 239.1.1.1

Join Multicast Group 239.2.2.2

Join Multicast Group 239.2.2.2Join Multicast Group

239.1.1.1


WebVM

WebVM

DBVM

DBVM

• Encapsulate with Blue VXLAN ID• Multicast to Servers Registered for 239.1.1.1

• Encapsulate with Red VXLAN ID• Multicast to Servers Registered for 239.2.2.2


VM 1 VM 2 VM 3

VXLAN VMKNIC1.1.1.1

VXLAN VMKNIC2.2.2.2

VXLAN VMKNIC3.3.3.3

MAC: abc

MAC: xyz

Multicast

VM1 Communicating with VM2 in a VXLAN

Multicast Multicast

VEM 1 VEM 2 VEM 3


VM 1 VM 2 VM 3

VXLAN VMKNIC1.1.1.1

VXLAN VMKNIC2.2.2.2

VXLAN VMKNIC3.3.3.3

MAC: abc

MAC: xyz

VM Source MAC Remote Host VXLAN IP

VM1:abc 1.1.1.1


Unicast

MAC Table: VEM 2

Layer 3


VM 1 VM 2 VM 3

VXLAN VMKNIC1.1.1.1

VXLAN VMKNIC2.2.2.2

VXLAN VMKNIC3.3.3.3

MAC: abc

MAC: xyz


VM1:abc 1.1.1.1

MAC Table: VEM 2


VM2:xyz 2.2.2.2

MAC Table: VEM 1


VEM 1 VEM 2 VEM 3


VM 1 VM 2 VM 3

VXLAN VMKNIC1.1.1.1

VXLAN VMKNIC2.2.2.2

VXLAN VMKNIC3.3.3.3

MAC: abc

MAC: xyz


Unicast


VM2:xyz 2.2.2.2

MAC Table: VEM 1


VM1:abc 1.1.1.1

MAC Table: VEM 2


Nexus 1000V VXLAN Integration with VMware vCloud Director


• Cisco Nexus 1000V Series 1.5 Release 4.2(1)SV1(5.2) is fully integrated into VMware vCloud Director

• Support dynamic network provisioningPort-group backed pools

VLAN-backed pools

Network isolation backed pools (via VXLAN)

• vSphere 4.1, 5.0, or 5.1

vCloud Director 1.5 or 5.1

vCentervShield Manager 5.0.1 or

5.1

vSphere 4.1, 5.0, or 5.1

Nexus 1000V v1.5.2

vShield Edge 5.0.1 or 5.1

Host


vCloud Director Integration

vCloud Director

vShield Manager

Network Services Mgr(Cisco Network Mgmt)

ASA 1000V(Security)

Nexus 1000V

vShield Edge(Security)

vSwitch

VMwareNetwork Stack

Cisco Network Stack(future)

VMware Cloud Orchestration

vShield Edge(Security)

VMware/Cisco Network Stack

Nexus 1000V

vSphere

Cisco Unified Computing System


vCloud Director Network Name

vSphere Port Group Name


VXLAN to VLAN Gateway

Nexus 1000V

REST API

Hypervisor

Tenant 1

Virtual Services

vWAAS

VSGASA 1KV

Tenant 3

ASA 55xx

Physical Workloads

Physical (VLAN) Network

VXLAN – VLANGateway

Virtual Workloads

Tenant 2

Nexus 1000V Quantum Plug-in

OpenStack


VXLAN to VLAN Gateway

Layer 3

WebVM

VXLAN GatewayVXLAN

Gateway

VXLAN GatewayVXLAN

Gateway

L2 Domain B L2 Domain CL2 Domain A

Bare MetalDB Server

VXLAN 5500

ASA5500

VLAN 100VLAN 200


Top 5 to Tell Network Admin @ VXLAN• IP Multicast forwarding is required (based on IETF draft)

More multicast groups are betterMultiple segments can be mapped to a single multicast groupIf VXLAN transport is contained to a single VLAN, IGMP Querier must be enabled on that VLANIf VXLAN transport is traversing routers, multicast routing must be enabled.

• Increased MTU needed to accommodate VXLAN encapsulation overheadPhysical infrastructure must carry 50 bytes more than the VM VNIC MTU size. e.g. 1500 MTU on VNIC -> 1550 MTU on switches and routers.

• Leverage 5-tuple hash distribution for uplink and interswitch LACP

• If VXLAN traffic is traversing a router, proxy ARP must be enabled on first hop router

• Prepare for more traffic between L2 domains


Summary• VXLAN is virtual overlay network for

multitenant cloud

• Nexus 1000V is first to support VXLAN and integrated with VMware vCloud Director

• VXLAN to VLAN Gateway provides virtual to physical connectivity

Top 5 for deploying VXLAN1. IP Multicast: Required

2. MTU Size: Increase 50 bytes

3. 5 Tuple Hashing: Turn on

4. Proxy ARP: For crossing L3 boundaries

5. More traffic between L2 domains

For More Information

http://tinyurl.com/N1k-Resources

Thank you.

Best Practices for Deploying VXLAN with Nexus 1000V and vCloud Director

Technology

Transcript of Best Practices for Deploying VXLAN with Nexus 1000V and vCloud Director