Best Practices for Virtual Networking: VMware, Inc.

37
© 2009 VMware Inc. All rights reserved Best Practices for Virtual Networking Karim Elatov Technical Support Engineer, GSS

Transcript of Best Practices for Virtual Networking: VMware, Inc.

Page 1: Best Practices for Virtual Networking: VMware, Inc.

© 2009 VMware Inc. All rights reserved

Best Practices for Virtual Networking

Karim Elatov

Technical Support Engineer, GSS

Page 2: Best Practices for Virtual Networking: VMware, Inc.

2

Agenda

Best Practices for Virtual Networking

Troubleshooting Virtual Networks

Virtual Network Overview

vSwitch Configurations

What’s New in vSphere 5.0

Tips & Tricks

Network Design Considerations

Page 3: Best Practices for Virtual Networking: VMware, Inc.

3

Virtu

al S

witc

h

Virtual

Conventional access, distribution, core design

Design with redundancy for enhanced availability

Under the covers, virtual network same as physical

Access layer implemented as virtual switches

Ph

ys

ica

l

Sw

itch

Ph

ys

ica

l

Sw

itch

Physical

Virtual Network Overview - Physical to Virtual

Physical

Page 4: Best Practices for Virtual Networking: VMware, Inc.

4

vNetwork Distributed

Switch

Distributed:

1 or more per

“Datacenter”

- Expanded feature set

- Private VLANs

- Bi-directional traffic shaping

- Network vMotion

- Simplified management

Virtual networking concepts similar with all virtual switches

Virtual Switch Options

Virtual Switch Model Details

vNetwork Standard

Switch

Host based:

1 or more per

ESX host

- Same as vSwitch in VI3

Cisco Nexus 1000V Distributed:

1 or more per

“Datacenter”

- Cisco Catalyst/Nexus feature set

- Cisco NXOS cli

- Supports LACP

Page 5: Best Practices for Virtual Networking: VMware, Inc.

5

ESX Virtual Switch: Capabilities

Layer 2 - only forward frames VM <-> VM and VM <-

> Uplink; No vSwitch <-> vSwitch or Uplink <-> Uplink

MAC

address

assigned to

vnic

VM0 VM1

vSwitch

Physical

Switches

vSwitch

MAC a MAC b MAC c

vSwitch will not create loops affecting Spanning

Tree in the physical network

Can terminate VLAN trunks (VST mode) or pass

trunk through to VM (VGT mode)

NIC Teaming of Physical NIC(s) [uplink(s)] associated

with vSwitches

Page 6: Best Practices for Virtual Networking: VMware, Inc.

6

Distributed Virtual Switch

vCenter vCenter

Standard vSwitch vNetwork & dvSwitch

Exist across 2 or more clustered hosts

•Provide similar functionality to vSwitches

•Reside on top of hidden vSwitches

vCenter owns the configuration of the dvSwitch

•Consistent host network configurations

Page 7: Best Practices for Virtual Networking: VMware, Inc.

7

Port Groups

Template for one or more ports with a common

configuration

• VLAN Assignment

• Security

• Traffic Shaping (limit egress traffic from VM)

• Failover & Load Balancing

Distributed Virtual Port Group (Distributed Virtual Switch)

• Bidirectional traffic shaping (ingress and egress)

• Network VMotion—network port state migrated upon

VMotion

Page 8: Best Practices for Virtual Networking: VMware, Inc.

8

NIC Teaming for Availability and Load Sharing

NIC Teaming aggregates multiple physical uplinks:

• Availability—reduce exposure to single points of

failure (NIC, uplink, physical switch)

• Load Sharing—distribute load over multiple

uplinks (according to selected NIC teaming

algorithm)

VM0 VM1

vSwitch

NIC Team

KB - NIC teaming in ESXi and ESX (1004088)

Requirements:

• Two or more NICs on same vSwitch

• Teamed NICs must have same VLAN configurations

Page 9: Best Practices for Virtual Networking: VMware, Inc.

9

NIC Teaming Options

Explicit Failover

Order

Highest order uplink

from active list

Teamed ports in same L2 domain

(BP: team over two physical

switches)

Best Practices:

•Originating Virtual PortID for VMs is the default, no extra configuration needed

•IP Hash, ensure that physical switch is properly configured for Etherchannel

*KB - ESX/ESXi host requirements for link aggregation (1001938)

*KB - Sample configuration of EtherChannel / Link aggregation with ESX/ESXi and Cisco/HP switches (1004048)

Name Algorithm—vmnic

chosen based upon:

Physical Network Considerations

Originating

Virtual Port ID

vnic port Teamed ports in same L2 domain

(BP: team over two physical

switches)

Source MAC

Address

MAC seen on vnic Teamed ports in same L2 domain

(BP: team over two physical

switches)

IP Hash* Hash(SrcIP, DstIP) Teamed ports configured in static

802.3ad “Etherchannel”

- no LACP (Nexus 1000v for LACP)

- Needs MEC to span 2 switches

Page 10: Best Practices for Virtual Networking: VMware, Inc.

10

Cisco Nexus 1000v Overview

Cisco Nexus 1000v is a software switch for vNetwork Distributed

Switches (vDS):

• Virtual Supervisor Module (VSM)

• Virtual Ethernet Module (VEM)

Things to remember:

• Virtual Ethernet Module (VEM)VSM uses external network fabric to

communicate with VEMs

• VSM does not take part in forwarding packets

• VEM does not switch traffic to other VEM without an uplink

Page 11: Best Practices for Virtual Networking: VMware, Inc.

11

Cisco Nexus 1000v Modules

vCenter Server

VMware ESX

Server 1

VMware vSwitch

VMware ESX

Server 2

VMware vSwitch

VMware ESX

Server 3

VMware vSwitch

VM #1

VM #4

VM #3

VM #2

VM #5

VM #8

VM #7

VM #6

VM #9

VM #12

VM #11

VM #10

Nexus 1000V

VSM

VEM VEM VEM Nexus 1000V vDS

Virtual Supervisor Module (VSM)

• Virtual or Physical appliance running Cisco OS (supports HA)

• Performs management, monitoring, & configuration

• Tight integration with VMware Virtual Center

Virtual Ethernet Module (VEM)

• Enables advanced networking capability on the hypervisor

• Provides each VM with dedicated “switch port”

• Collection of VEMs = 1 DVS

Cisco Nexus 1000V Enables:

• Policy Based VM Connectivity

• Mobility of Network & Security Properties

• Non-Disruptive Operational Model

Page 12: Best Practices for Virtual Networking: VMware, Inc.

12

vSwitch Configurations

Best Practices for Virtual Networking

Troubleshooting Virtual Networks

Virtual Network Overview

vSwitch Configurations

What’s New in vSphere 5.0

Tips & Tricks

Network Design Considerations

Page 13: Best Practices for Virtual Networking: VMware, Inc.

13

Cisco ‘show run’ and ‘show tech-support’

The following is a Cisco EtherChannel sample configuration: interface Port-channel1 switchport switchport access vlan 100 switchport mode access no ip address ! interface GigabitEthernet1/1 switchport switchport access vlan 100 switchport mode access no ip address channel-group 1 mode on !

Obtain configuration of a Cisco router or switch

•Run commands in priviliged EXEC mode

•’show run’

•‘show tech-support’

KB - Troubleshooting network issues with the Cisco show tech-support command (1015437)

Page 14: Best Practices for Virtual Networking: VMware, Inc.

14

Traffic Types on a Virtual Network

Virtual Machine Traffic

• Traffic sourced and received from virtual machine(s)

• Isolate from each other based on service level

How do we maintain traffic isolation without proliferating NICs? VLANs

vMotion Traffic

• Traffic sent when moving a virtual machine from one ESX host to another

• Should be isolated

Management Traffic

• Should be isolated from VM traffic (one or two Service Consoles)

• If VMware HA is enabled, includes heartbeats

IP Storage Traffic—NFS and/or iSCSI via vmkernel interface

• Should be isolated from other traffic types

Fault Tolerance (FT) Logging Traffic

• Low latency, high bandwidth

• Should be isolated from other traffic types

Page 15: Best Practices for Virtual Networking: VMware, Inc.

15

Traffic Types on a Virtual Network, cont.

Port groups in dedicated VLANs on a management-only virtual

switch.

vMotion

production

virtual switch

Service console/VMK Interface

virtual machines vMotion

106

storage

107

mgmt

108 management

virtual switch

production management storage

Page 16: Best Practices for Virtual Networking: VMware, Inc.

16

VLAN Tagging Options

vSwitch

Physical Switch

VST – Virtual Switch Tagging

VLAN Tags

applied in

vSwitch

VST is the best practice and

most common method

VLAN

assigned in

Port Group

policy

vSwitch

Physical Switch

EST – External Switch Tagging

External Physical

switch applies

VLAN tags

switchport access vlan switchport trunk

vSwitch

Physical Switch

VGT – Virtual Guest Tagging

VLAN Tags

applied in

Guest

PortGroup

set to VLAN

“4095”

switchport trunk

Page 17: Best Practices for Virtual Networking: VMware, Inc.

17

DVS Support for Private VLAN (PVLAN)

Enable users to restrict communications

• Between VMs on the same VLAN or network segment

PVLAN Types

• Community

• VMs can communicate with VMs on Community and Promiscuous

• Isolated

• VMs can only communicate with VMs on the Promiscuous

• Promiscuous

• VMs can communicate with all VMs

KB - Private VLAN (PVLAN) on vNetwork Distributed Switch - Concept Overview (1010691)

Allow devices to share the same IP subnet while being Layer 2 Isolated

Benefits:

•Employ Larger subnets (advantageous to hosting environments)

•Reduce Management Overhead

application

server

Web

server database

server

email

server document

server

isolated

PVLAN

isolated

PVLAN community PVLAN

DMZ network

router in promiscuous PVLAN

Page 18: Best Practices for Virtual Networking: VMware, Inc.

18

W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B

Distributed Virtual Switch

PG PG PG PG PG PG PG PG PG PG PG PG

TOTAL COST: 12 VLANs (one per VM)

TOTAL COST: 1 PVLAN (over 90% savings…)

W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B

Distributed Virtual Switch

PG (with Isolated PVLAN)

PVLAN Cost Benefit

Page 19: Best Practices for Virtual Networking: VMware, Inc.

19

Link Aggregation

EtherChannel

•Port trunking between two to eight

•Active Fast Ethernet, Gigabit Ethernet, or 10 Gigabit Ethernet ports

KB ESX/ESXi host requirements for link aggregation (1001938)

LACP (one of the implementations included in IEEE 802.3ad)

•Link Aggregation Control Protocol (LACP)

•Control the bundling of several physical ports into a single logical channel

•Only supported on Nexus 1000v

EtherChannel vs. 802.3ad

•EtherChannel is Cisco proprietary and 802.3ad is an open standard

Note: ESX implements 802.3ad Static Mode Link Aggregation

Page 20: Best Practices for Virtual Networking: VMware, Inc.

20

Sample Link Aggregation Configuration

KB - Sample configuration of EtherChannel / Link aggregation with ESX/ESXi andCisco/HP switches (1004048)

Supported switch Aggregation algorithm: IP-SRC-DST Supported Virtual Switch NIC Teaming mode: IP HASH

Page 21: Best Practices for Virtual Networking: VMware, Inc.

21

Failover Configurations

Figure — Using beacons to detect upstream

network connection failures. KB - What is beacon probing? (1005577)

Beacon Probing sends out and listens for beacon probes

•Broadcast frames (ethertype 0x05ff)

Beacon Probing Best Practice

•Use at least 3 NICs for triangulation

•If only 2 NICs in team, can’t determine link failed

•Leads to shotgun mode results

Link Status relies solely on the network adapter link state

•Cannot detect configuration errors

•Spanning Tree Blocking

•Incorrect VLAN

•Physical switch cable pulls

Page 22: Best Practices for Virtual Networking: VMware, Inc.

22

Spanning Tree Protocol (STP) Considerations

Spanning Tree Protocol creates loop-free L2 tree topologies in the physical network

• Physical links put in “blocking” state to construct loop-free tree

VM0 VM1

vSwitch

Physical

Switches

MAC a MAC b

Switches sending

BPDUs every 2s to

construct and

maintain Spanning

Tree Topology

vSwitch drops

BPDUs

Blocked link

Recommendations for Physical Network Config: 1. Leave Spanning Tree enabled on physical network

and ESX facing ports (i.e. leave it as is!) 2. Use “portfast” or “portfast trunk” on ESX facing

ports (puts ports in forwarding state immediately) 3. Use “bpduguard” to enforce STP boundary

KB - STP may cause temporary loss of network connectivity when a failover or failback event occurs (1003804)

ESX vSwitch does not participate in Spanning Tree and will not create loops with uplinks

•ESX Uplinks will not block, always active (full use of all links)

Page 23: Best Practices for Virtual Networking: VMware, Inc.

23

Tips & Tricks

Best Practices for Virtual Networking

Troubleshooting Virtual Networks

Virtual Network Overview

vSwitch Configurations

What’s New in vSphere 5.0

Tips & Tricks

Network Design Considerations

Page 24: Best Practices for Virtual Networking: VMware, Inc.

24

Tips & Tricks

Load-Based Teaming (LBT)

• Dynamically balance network load over available uplinks

• Triggered by ingress or egress congestion at 75% mean utilization over a 30

second period

• Configure on DVS via “Route based on physical NIC load”

*LBT is not available on the Standard vSwitch (DVS feature for ingress/egress traffic shaping)

Network I/O Control (NetIOC)

• DVS software scheduler to isolate and prioritize specific traffic types

contending for bandwidth on the uplinks connecting ESX/ESXi 4.1 hosts with

the physical network.

Page 25: Best Practices for Virtual Networking: VMware, Inc.

25

Tips & Tricks

Tip #5 – Link aggregation is never supported on disparate trunked switches – Use

VSS with MEC. (KB 1001938 & KB 1027731)

Tip #4 - Beacon Probing and IP Hash DO NOT MIX (duplicate packets and port flapping) (KB 1017612 & KB 1012819)

Tip #1 – After physical to virtual migration, the VM MAC address can be changed for Licensed Applications relying on physical MAC address. (KB 1008473)

Tip #2 – NLB Multicast needs physical switch Manual ARP resolution of NLB cluster. (KB 1006525)

Tip #3 – Cisco Discovery Protocol (CDP) gives switchport configuration information useful for troubleshooting (KB 1007069)

Page 26: Best Practices for Virtual Networking: VMware, Inc.

26

Using 10GigE

2x 10GigE common/expected

• 10GigE CNAs or NICs

Possible Deployment Method

• Active/Standby on all Portgroups

• VMs “sticky” to one vmnic

• SC/vmk ports sticky to other

• Use Ingress Traffic Shaping

to control traffic type per

Port Group

• If FCoE, use Priority Group

bandwidth reservation (on CNA

utility)

vSwitch

iSCSI NFS VMotion FT SC

FCoE FCoE

SC#2

FCoE

10

FCoE Priority Group

bandwidth reservation

(in CNA config utility)

Gbps 10GE 10GE

Ingress (into switch)

traffic shaping policy

control on Port Group

1-2G Low b/w High

b/w

Variable/high

b/w 2Gbps+

Tips & Tricks

Best Practice: Ensure Drivers and Firmware are compatible for success

vSphere 4.1 supports up to (4) 10GigE NICs; 5.0 supports (8) 10GigE NICs

Page 27: Best Practices for Virtual Networking: VMware, Inc.

27

Troubleshooting Virtual Networks

Best Practices for Virtual Networking

Troubleshooting Virtual Networks

Virtual Network Overview

vSwitch Configurations

What’s New in vSphere 5.0

Tips & Tricks

Network Design Considerations

Page 28: Best Practices for Virtual Networking: VMware, Inc.

28

Network Troubleshooting Tips

Troubleshoot one component at a time

• Physical NICs

• Virtual Switch

• Virtual NICs

• Physical Network

Tools for Troubleshooting

• vSphere Client

• Command Line Utilities

• ESXTOP

• Third party tools

• Ping and Traceroute

• Traffic sniffers & Protocol Analyzers

• Wireshark

• Logs

Page 29: Best Practices for Virtual Networking: VMware, Inc.

29

Capturing Traffic

ESXi uses tcpdump-uw (KB 1031186)

vSwitch must be in Promiscuous Mode (KBs 1004099 & 1002934)

Best Practice: create a new management interface for this purpose

Page 30: Best Practices for Virtual Networking: VMware, Inc.

30

What’s New in vSphere 5.0

Best Practices for Virtual Networking

Troubleshooting Virtual Networks

Virtual Network Overview

vSwitch Configurations

What’s New in vSphere 5.0

Tips & Tricks

Network Design Considerations

Page 31: Best Practices for Virtual Networking: VMware, Inc.

31

What’s New in vSphere 5?

Monitor and troubleshoot virtual infrastructure traffic

• NetFlow V5

• Port mirror (SPAN)

• LLDP (standard based link layer discovery protocol) support simplifies the

network configuration and management in non-Cisco switch environment.

Enhancements to the network I/O control (NIOC)

• Ability to create User-defined resource pool

• Support for vSphere replication traffic type; a new system traffic type that

carries replication traffic from one host to another.

• Support for IEEE 802.1p tagging

What’s New in VMware vSphere 5.0 Networking Technical Whitepaper

Page 32: Best Practices for Virtual Networking: VMware, Inc.

32

Network Design Considerations

Best Practices for Virtual Networking

Troubleshooting Virtual Networks

Virtual Network Overview

vSwitch Configurations

What’s New in vSphere 5.0

Tips & Tricks

Network Design Considerations

Page 33: Best Practices for Virtual Networking: VMware, Inc.

33

Network Design Considerations

How do you design the virtual network for performance and availability but maintain isolation between the various traffic types (e.g. VM traffic, VMotion, and Management)?

• 2 NIC minimum for availability, 4+ NICs

per server preferred

• 802.1Q VLAN trunking highly recommended for logical scaling

(particularly with low NIC port servers)

• Examples are meant as guidance and do not represent strict

requirements in terms of design

• Understand your requirements and resultant traffic types and

design accordingly

• Starting point depends on:

• Number of available physical ports on server

• Required traffic types

Page 34: Best Practices for Virtual Networking: VMware, Inc.

34

Candidate Design:

• Team both NIC ports

• Create one virtual switch

• Create three port groups:

• Use Active/Standby policy

for each portgroup

• Portgroup1: Service Console (SC)

• Portgroup2: VMotion

• Portgroup3: VM traffic

• Use VLAN trunking

• Trunk VLANs 10, 20,

30 on each uplink

vmnic0

Active

Standby

vmnic1

Portgroup3

VLAN 30

VLAN Trunks

(VLANs 10, 20, 30)

Note: Team over dvUplinks with vDS

Portgroup1

VLAN 10

Portgroup2

VLAN 20

SC vmkernel

vSwitch

Example 1: Blade Server with 2 NIC Ports

Page 35: Best Practices for Virtual Networking: VMware, Inc.

35

Candidate Design:

• Create two virtual switches

• Team two NICs to each vSwitch

• vSwitch0 (use active/standby

for each portgroup):

• Portgroup1: Service Console (SC)

• Portgroup2: VMotion

• vSwitch1 (use Originating Virtual

PortID)

• Portgroup3: VM traffic #1

• Portgroup4: VM traffic #2

• Use VLAN trunking

• vmnic1 and vmnic3: Trunk VLANs 10, 20

• vmnic0 and vmnic2: Trunk VLANs 30, 40

VLANs

10, 20

vSwitch0 vSwitch1

VLANs

30, 40

Note: Team over dvUplinks with vDS

Active

Standby

SC vmkernel

Portgroup1

VLAN 10

Portgroup2

VLAN 20

vmnic0 vmnic1 vmnic3

Portgroup3

VLAN 30

Portgroup4

VLAN 40

vmnic2

Example 2: Server with 4 NIC Ports

Page 36: Best Practices for Virtual Networking: VMware, Inc.

36

Candidate Design:

• Create one virtual switch

• Create two NIC teams

• vSwitch0 (use active/standby

for portgroups 1 & 2):

• Portgroup1: Service Console (SC)

• Portgroup2: Vmotion

• Use Originating Virtual PortID

for Portgroups 3 & 4

• Portgroup3: VM traffic #1

• Portgroup4: VM traffic #2

• Use VLAN trunking

• vmnic1 and vmnic3: Trunk VLANs 10, 20

• vmnic0 and vmnic2: Trunk VLANs 30, 40

vmnic0

SC vmkernel

vmnic1 vmnic2 vmnic3

Active

Standby

Note: Team over dvUplinks with vDS

Portgroup1

VLAN 10

Portgroup2

VLAN 20

Portgroup3

VLAN 30

Portgroup4

VLAN 40

VLANs

10, 20

VLANs

30, 40

vSwitch0

Example 3: Server with 4 NIC Ports (Slight Variation)

Page 37: Best Practices for Virtual Networking: VMware, Inc.

37

Questions