VMworld 2013: Operational Best Practices for NSX in VMware Environments

67
Operational Best Practices for NSX in VMware Environments Ray Budavari, VMware Thomas Kraus, VMware NET5790 #NET5790

description

VMworld 2013 Ben Basler, VMware Roberto Mari, VMware Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare

Transcript of VMworld 2013: Operational Best Practices for NSX in VMware Environments

Page 1: VMworld 2013: Operational Best Practices for NSX in VMware Environments

Operational Best Practices for NSX in VMware

Environments

Ray Budavari, VMware

Thomas Kraus, VMware

NET5790

#NET5790

Page 2: VMworld 2013: Operational Best Practices for NSX in VMware Environments

2 2

Agenda

Introduction - Network Virtualization

Operational Impacts

NSX for vSphere Components

Operational Tools

Demonstrations

Conclusion

Page 3: VMworld 2013: Operational Best Practices for NSX in VMware Environments

3 3

Introduction - Network Virtualization

1. Decouple

Physical

Virtual

2. Reproduce 3. Automate

Network

Operations

Cloud

Operations

Hardware

independence

Operational benefits

of virtualization No change to network

from end host perspective

Virtual

Physical

Page 4: VMworld 2013: Operational Best Practices for NSX in VMware Environments

4 4

Agenda

Introduction - Network Virtualization

Operational Impacts

NSX for vSphere Components

Operational Tools

Demonstrations

Conclusion

Page 5: VMworld 2013: Operational Best Practices for NSX in VMware Environments

5 5

Operational Impacts - Questions

If a Virtual Machine has a network outage where do I start?

How does network virtualization map to our operating model?

What tools exist to correlate logical and physical networks to

assist in troubleshooting?

What opportunities does network virtualization provide to improve

how we operate our environment?

Are we adding complexity

by adopting network

virtualization?

Does network virtualization

create a ‘black box’?

Page 6: VMworld 2013: Operational Best Practices for NSX in VMware Environments

6 6

Operational Impacts - Answers

Capability Physical Virtual Capability Physical Virtual

Packet Capture ✔ ✔ VM level

visibility ✗ ✔ NetFlow

✔ ✔ Network

Snapshot ✗ ✔ RSPAN/ERSPA

N ✔ ✔ CLI ✔ ✔ Performance

Statistics ? ✔ UI ? ✔ Syslog ✔ ✔ API ? ✔

Page 7: VMworld 2013: Operational Best Practices for NSX in VMware Environments

7 7

Operational Impacts - Opportunities

All NSX components such as the NSX Controller, NSX vSwitch and

NSX Edge provide detailed network visibility and data

Simplify the underlying physical network

• One consistent physical transport network to manage for virtual machine traffic

• Greatly reduces the number of MAC/ARP table entries to manage

• Enables you to build the network you want, while still meeting application and

workload connectivity requirements

DC Networks

Centralized reporting and

monitoring, distributed

performance and scale

Designed for automation

• NSX is built on a REST API

provided by NSX Manager

• All operations can be performed

programmatically via scripting or

higher-level languages

Page 8: VMworld 2013: Operational Best Practices for NSX in VMware Environments

8 8

Operational Impacts - Examples

If a Virtual Machine has a network outage where do I start to troubleshoot?

Before Network Virtualization:

• Validate VLAN trunk configuration across multiple devices and ports

• Verify VM visibility on each path of the network

• Troubleshooting requires accessing different devices and interfaces

• vSphere Web Client

• Hypervisor CLI

• Access Switch CLI

• Distribution Switch CLI

• Firewall

• Load Balancer

• These devices are typically managed by different teams

• Virtualization Administrators

• Network Administrators

• Security Administrators

• Service Providers

Page 9: VMworld 2013: Operational Best Practices for NSX in VMware Environments

9 9

Operational Impacts - Examples

After Network Virtualization:

• VXLAN network tests determine if the issue is related to the transport network

• If VTEPs are reporting issues, engage the network team to troubleshoot physical

transport network

• Provide VTEP IP/MAC information

• Otherwise virtualization team validates VM logical networking

• Verify NSX Components and Controller state information

• Verify Source and Destination Hypervisors

• Enhanced toolset is available for troubleshooting

• Reduced number of components and resources required

• NSX components can be queried or configured via REST API

Page 10: VMworld 2013: Operational Best Practices for NSX in VMware Environments

10 10

Agenda

Introduction - Network Virtualization

Operational Impacts

NSX for vSphere Components

Operational Tools

Demonstrations

Conclusion

Page 11: VMworld 2013: Operational Best Practices for NSX in VMware Environments

11 11

NSX for vSphere Components

Consumption

• Self Service Portal

• Cloud Management

• vCloud Automation Center

Data

Plane

NSX Edge

Services

Gateway

ESXi

VDS

Hypervisor Kernel Modules

Firewall Distributed

Logical Router

VXLAN

NSX vSwitch

• NSX Edge

• VM form factor

• Data Plane for North South

traffic

• Routing and Advanced

services

• NSX vSwitch

• Distributed network edge

• Line Rate performance

Management

Plane

NSX Manager • Single point of configuration

• REST API and UI interface

vCenter Server

Control

Plane

NSX Controller • Manages Logical networks

• Run-time state

• Does not sit in the Data Path

• Control-Plane Protocol

NSX Edge

Logical Router User World Agent

Page 12: VMworld 2013: Operational Best Practices for NSX in VMware Environments

12 12

Components – NSX Manager

• NSX for vSphere centralized management plane

• 1:1 mapping between an NSX Manager and vCenter Server

• Provides the management UI and API for NSX

• vSphere Web Client Plugin

• Deploys NSX Controller and NSX Edge Virtual Appliances (OVF)

• Installs VXLAN, Distributed Routing and Firewall kernel modules and UW

Agent on ESXi hosts

• Configures Controller

Cluster via a REST API

and hosts via a

message bus

• Generates certificates to

secure control plane

communications

Page 13: VMworld 2013: Operational Best Practices for NSX in VMware Environments

13 13

Components – NSX Controller

A reliable and secure control plane to distribute VXLAN and Logical

Routing network information to ESXi hosts

NSX Controllers are clustered for scale out and high availability

Network information is sliced across nodes in a Controller Cluster

Enables dependency on multicast routing/PIM in the physical network to

be removed

Provides suppression of ARP broadcast traffic in VXLAN networks

VXLAN

Logical Router

VXLAN

Logical Router

VXLAN

Logical Router

Controller

VXLAN Directory

Service

MAC table

ARP table

VTEP table

Page 14: VMworld 2013: Operational Best Practices for NSX in VMware Environments

14 14

Components – UW agent

UW agent is a TCP (SSL) client that communicates with the Controller using

the control plane protocol

May connect to multiple controllers

Mediator between the ESXi Hypervisor Kernel Modules and NSX Controllers

Also communicates with message bus agent to retrieve information from|

NSX Manager

Runs as a service daemon on ESXi: netcpa

• Logs to: /var/log/netcpa.log

Controller

Cluster Controller Controller Controller

ESXi Host

Kernel

Modules

Client Client User World

Agent

LR

NSX

MGR

Client

VXLAN

Page 15: VMworld 2013: Operational Best Practices for NSX in VMware Environments

15 15

Components – NSX vSwitch and NSX Edge

NSX vSwitch (VDS)

VMkernel Modules

VXLAN

Distributed Routing

Distributed Firewall

Switch Security

Message Bus

L3-L7 Services:

NAT, DHCP, LB, VPN, Interface based

FW

Dynamic Routing

VM form factor

High Availability

vSphere NSX Edge Services GW

Control Functions only

Dynamic Routing &

updates to Controller

Determines active ESXi

host for L2 Bridging

NSX Edge Logical Router

ESXi

VDS

Hypervisor Kernel Modules

(vSphere VIBs)

Firewall Logical Router VXLAN

NSX vSwitch

NSX Edge

Services

Gateway

NSX Edge

Logical

Router

Page 16: VMworld 2013: Operational Best Practices for NSX in VMware Environments

16 16

Agenda

Introduction - Network Virtualization

Operational Impacts

NSX for vSphere Components

Operational Tools

Demonstrations

Conclusion

Page 17: VMworld 2013: Operational Best Practices for NSX in VMware Environments

17 17

Operational Tools – ESXi

pktcap-uw

New with vSphere 5.5

Enhanced tool that provides a framework for packet capture and tracing at the

Uplink, vSwitch, vmknic, vnic and port level at any stage in a packet’s lifecycle

Page 18: VMworld 2013: Operational Best Practices for NSX in VMware Environments

18 18

Operational Tools – ESXi

pktcap-uw

Extensive range of filters

such as source/destination

mac, IP, Protocol, VLAN,

VXLAN, ports etc.

Supports pcap format

output for use with protocol

analyzers such as

Wireshark

Page 19: VMworld 2013: Operational Best Practices for NSX in VMware Environments

19 19

Operational Tools – NSX vSwitch Backup & Restore

vSwitch Backup & Restore

Page 20: VMworld 2013: Operational Best Practices for NSX in VMware Environments

20 20

Operational Tools – NSX vSwitch Netflow

NetFlow / IPFIX

NetFlow collector

address and port

Page 21: VMworld 2013: Operational Best Practices for NSX in VMware Environments

21 21

Operational Tools – NSX vSwitch RSPAN/ERSPAN

RSPAN/ERSPAN, Port Mirroring

Page 22: VMworld 2013: Operational Best Practices for NSX in VMware Environments

22 22

Operational Tools – NSX vSwitch Alarms

vSwitch Alarms allow for alerting on VDS related events. Some of the

available preconfigured Triggers include Health Status, Reconfiguration,

Port blocked, Port Deleted, Link Down and Host removal

SNMP Network MIBs provide standards based visibility of NSX vSwitch

objects

Page 23: VMworld 2013: Operational Best Practices for NSX in VMware Environments

23 23

Operational Tools – NSX vSwitch Health Check

Network Health Check feature helps to detect common configuration errors

Mismatched VLAN trunks between virtual switch and physical switch

Mismatched MTU setting between vNIC, virtual switch, physical adapter, and physical switch ports.

Mismatched Teaming Configurations

vSphere admins can provide failure data to the Network admins to facilitate problem resolution

Health Check uses

L2 Echo protocol to

send Ethernet

broadcast frames to

the physical switch

If reply packets are

not received,

warnings are

highlighted in the

vSphere Web Client

Page 24: VMworld 2013: Operational Best Practices for NSX in VMware Environments

24 24

vSphere Cluster B

UWA VTEP

UWA VTEP

UWA VTEP

Operational Tools – NSX Controller

Control Plane basics

ESXi hosts and NSX Edge Logical

Router VMs collect network

information, which is then reported

to the Controller via User World

Agent (UWA)

The NSX Controller CLI provides a

consistent, centralized interface to

verify VXLAN and Logical Routing

network state information

NSX Manager also provides APIs to

programmatically retrieve data from

the controller nodes

NSX Manager

NSX

Controller

Cluster

vSphere Cluster A

UWA VTEP

UWA VTEP

UWA VTEP

Page 25: VMworld 2013: Operational Best Practices for NSX in VMware Environments

25 25

Operational Tools – NSX Controller VTEP Report

vSphere Host

VM

vSphere Distributed Switch

MAC1

Management

Network

10.20.10.10

vSphere Host

VM

MAC2

vSphere Host

Send VNI,VTEP

Mapping to

Controller

1

2

5

10

VXLAN 5001

3

VNI VTEP IP

5001 10.20.10.10

9

IP1 IP

2

Controller

VNI VTEP IP

5001 10.20.10.10

VNI VTEP IP

5001 10.20.10.11

10.20.10.11 10.20.10.12

11

6 7

VNI VTEP IP

5001 10.20.10.10

10.20.10.11

VNI VTEP IP

5001 10.20.10.10

10.20.10.11

4 8

10

VNI VTEP IP

5001 10.20.10.10

10.20.10.11

11

Report the new

VNI,VTEP

Mapping to the

Hosts

Page 26: VMworld 2013: Operational Best Practices for NSX in VMware Environments

26 26

Operational Tools – NSX Controller

General NSX Controller troubleshooting steps:

• Verify Controller cluster status and roles

• Verify Controller node network connectivity

• Check Controller API service

• Validate VXLAN and Logical Router mapping table entries to ensure they are consistent

• Review source and destination netcpa logs and CLI to determine control plane connectivity issues between ESXi hosts & NSX Controller

The first set of commands relates to NSX Controller CLI cluster status and health:

• nsx-controller # show control-cluster status

• nsx-controller # show control-cluster startup-nodes

• nsx-controller # show control-cluster roles

• nsx-controller # show control-cluster connections

• nsx-controller # show control-cluster vnet core stats

• nsx-controller # show network <arg>

• nsx-controller # show log cloudnet/cloudnet_java-vnet-controller.<start-time-stamp>.log

Page 27: VMworld 2013: Operational Best Practices for NSX in VMware Environments

27 27

Operational Tools – NSX Controller

General NSX Controller VXLAN operations:

• List VNIs

• VXLAN connection table

• Verify VXLAN VTEP, MAC and ARP mapping tables

• View VXLAN statistics

NSX Controller CLI VXLAN commands:

• # show control-cluster vnet vxlan vni <vni>

• # show control-cluster vnet vxlan connection-table <vni>

• # show control-cluster vnet vxlan vtep-table <vni>

• # show control-cluster vnet vxlan mac-table <vni>

• # show control-cluster vnet vxlan arp-table <vni>

• # show control-cluster vnet vxlan vni-stats <vni>

Note: VXLAN Logical Switches and Logical Router instances are distributed across Controller Nodes (slicing), so you will need to run the CLI commands on the node which is active for a given object

Page 28: VMworld 2013: Operational Best Practices for NSX in VMware Environments

28 28

Operational Tools – NSX Controller

General NSX Controller Logical Routing operations:

• List Logical Router instances

• Verify Logical Router interface and route mapping tables

• Verify active controller connections

• View Logical Router statistics

NSX Controller CLI Logical Routing commands:

• # show control-cluster vnet logical-router instance-table <lr-id>

• # show control-cluster vnet logical-router lif-table <lr-id>

• # show control-cluster vnet logical-router route-table <lr-id>

• # show control-cluster vnet logical-router lr-stats <lr-id>

Note that the NSX Controller CLI is still not final

Page 29: VMworld 2013: Operational Best Practices for NSX in VMware Environments

29 29

Operational Tools – VXLAN

Common VXLAN issues:

• Connectivity between VXLAN VTEPs on Transport Network

• MTU on Transport Network not set to 1600 bytes or greater

• Teaming mismatch between dvUplinks and upstream switch

• Preparation - either with the installation of VXLAN kernel modules or creation of VTEP VMkernel interfaces

• ESXi host communication with Controller

NSX for vSphere provides a new workflow for preparation and configuration

Supports multiple VTEPs per host

NSX leverages the vSphere 5.5 multi-instance TCP/IP stack

Page 30: VMworld 2013: Operational Best Practices for NSX in VMware Environments

30 30

Operational Tools – VXLAN

At Logical Switch level on the Monitoring Tab you can use the

unicast or broadcast test to verify the connectivity between VTEPs

VXLAN standard tests

with a 1600 byte MTU

Page 31: VMworld 2013: Operational Best Practices for NSX in VMware Environments

31 31

Operational Tools – VXLAN

The VXLAN Replication Mode will also determine transport network

connectivity requirements

Unicast Mode

• All replication occurs using

unicast

Hybrid Mode

• Local replication offloaded to

physical network, while remote

replication occurs via unicast

Multicast Mode

• Requires IGMP for a Layer 2

topology and Multicast Routing

for L3 topology

All modes require an MTU of

1600 bytes

Page 32: VMworld 2013: Operational Best Practices for NSX in VMware Environments

32 32

Operational Tools – VXLAN

VXLAN namespace for esxcli provides detailed network information and statistics. • # esxcli network vswitch dvs vmware vxlan list

• # esxcli network vswitch dvs vmware vxlan network list --vds-name=Compute_VDS

• # esxcli network vswitch dvs vmware vxlan network mac list –vds-name=Compute_VDS --vxlan-id=5001

• # esxcli network vswitch dvs vmware vxlan network arp list --vds-name Compute_VDS --vxlan-id=5001

• # esxcli network vswitch dvs vmware vxlan network port list --vds-name Compute_VDS --vxlan-id=5001

• # esxcli network vswitch dvs vmware vxlan network stats list --vds-name Compute_VDS --vxlan-id=5001

Page 33: VMworld 2013: Operational Best Practices for NSX in VMware Environments

33 33

Operational Tools – Logical Routing

Use the net-vdr command on ESXi hosts to view Logical Routing configuration and statistics

• Display Logical Router instances ~ # net-vdr -I –l

• List Logical Interface and Routing Tables ~ # net-vdr -l –lif <instance-name> ~ # net-vdr -l --route <instance-name>

LIFs and routes are pushed by the NSX controller to the ESXi hosts and should be consistent across the environment

• View L2 Bridging information ~ # net-vdr -b –mac <instance-name> ~ # net-vdr -l –stats <instance-name>

Page 34: VMworld 2013: Operational Best Practices for NSX in VMware Environments

34 34

Operational Tools – NSX Edge Services

NSX Edge VM CLIs

• NSX Edge provides a familiar CLI command set for troubleshooting network services

• Documented in a dedicated CLI guide

Sample Configuration Commands

• show configuration {ospf|bgp|isis|static-routing}

• show configuration {firewall|nat|dhcp|dns}

• show configuration {loadbalancer|ipec|sslvpn-plus}

Sample Status Commands

• show interface [IFNAME]

• show firewall

• show ip {route|ospf|bgp|forwarding}

• show arp

• show system {cpu|memory|network-stats|storage|uptime}

• show service {dhcp|dns|highavailability|ipsec|loadbalancer|sslvpn-plus}

Page 35: VMworld 2013: Operational Best Practices for NSX in VMware Environments

35 35

Operational Tools – NSX Edge Services

API provides based statistics (for interfaces and services)

UI also provides interface statistics and graphs

Page 36: VMworld 2013: Operational Best Practices for NSX in VMware Environments

36 36

Operational Tools – NSX Edge Services

Logging commands

• show log {follow|reverse}

• show flowtable

Debug/troubleshooting commands

• traceroute <ip_address or dns_name>

• ping <ip address> or ping interface addr <alternate_src_ip> <ip_address>

• debug packet display interface <vNic_0-9> <EXPRESSION>

• debug packet display interface vNic_0 host_192.168.1.2

• debug packet display interface vNic_2 host_192.168.1.3_and_port_80

• debug packet display interface vNic_1 src_192.168.1.2_and_dst_192.168.1.3

• debug packet capture interface <vNic_0-9> <EXPRESSION>

• debug show files

• debug copy {scp|ftp} <URL>

debug messagebus {forwarder|messages}

Page 37: VMworld 2013: Operational Best Practices for NSX in VMware Environments

37 37

Operational Tools – Flow Monitoring

Flow monitoring provides vNIC level visibility of VM traffic flows

Reporting on Top Flows, Destinations and Sources

Detailed Flow Data for both Allowed and Blocked Flows

Page 38: VMworld 2013: Operational Best Practices for NSX in VMware Environments

38 38

Operational Tools – Flow Monitoring

Flow data easily available through UI or via API for orchestration

Per flow granularity for Allowed and Blocked with ability to add or

edit firewall rules related to the flow.

Page 39: VMworld 2013: Operational Best Practices for NSX in VMware Environments

39 39

Operational Tools – NSX Manager

Perform Backup & Restore

operations (both scheduled

and on-demand)

NSX Manager Appliance

Upgrades

Verify status of NSX

Manager Services

Generate Tech support logs

View appliance CPU,

Memory and Disk usage

Page 40: VMworld 2013: Operational Best Practices for NSX in VMware Environments

40 40

Operational Practices – vCenter Operations Manager

Dashboard based view of environment

Monitor overall health of NSX vSphere Management and Control

Layer components and diagnose issues quickly

Networking and Security metrics

Page 41: VMworld 2013: Operational Best Practices for NSX in VMware Environments

41 41

Operational Tools – Syslog

Syslog is supported across all NSX components

• 1) NSX Manager

• 2) NSX Controllers

• 3) NSX Edges

• 4) ESXi Hosts

1

2

3

4

Page 42: VMworld 2013: Operational Best Practices for NSX in VMware Environments

42 42

Operational Practices – Log Insight

Consolidation, visualization, and correlation of syslog data from

multiple related components in a Software Defined Datacenter

Build Custom Dashboards for real time monitoring and trending

Customize Log interpretation Logic to parse using regex, int, str

Page 43: VMworld 2013: Operational Best Practices for NSX in VMware Environments

43 43

Operational Tools – REST API

NSX Manager exposes web service API over HTTPS (TCP 443)

API request and response data is formatted in XML

Simple “single-user” authentication using password

REST principles:

• Leverages HTTP to send data between Clients and Servers (Requests and Responses)

• Resources, Global Permanent Identifiers, Constraints

Page 44: VMworld 2013: Operational Best Practices for NSX in VMware Environments

44 44

Agenda

NSX Platform / Network Virtualization

Operational Impacts

NSX for vSphere Components

Operational Tools

Demonstrations

Conclusion

Page 45: VMworld 2013: Operational Best Practices for NSX in VMware Environments

45 45

NSX for vSphere Demonstrations

1. Packet capture of encapsulated VXLAN frames

2. Flow Monitoring

Page 46: VMworld 2013: Operational Best Practices for NSX in VMware Environments

46 46

Key Takeaways & Best Practices

VMware NSX provides a unified platform for administering,

monitoring, and supporting your virtual networks and services

NSX enables a similar operational model for virtual networks as

vSphere does for virtual machines

Moving network features to logical space simplifies physical

networks and troubleshooting

Start with the basics when troubleshooting (transport network and

control plane)

Understanding the component interactions and toolset is key to

NSX operations

Enable logging on all components ‘before’ you have issues and

familiarize yourself with how to collect support logs

Automate repeatable steps via the REST API to reduce error

Take the NSX for vSphere Hands on Lab: HOL-SDC-1303 to

reinforce concepts from this session

Page 47: VMworld 2013: Operational Best Practices for NSX in VMware Environments

47 47

Questions

Page 48: VMworld 2013: Operational Best Practices for NSX in VMware Environments

48 48

Backup Slides

Page 49: VMworld 2013: Operational Best Practices for NSX in VMware Environments

49 49

Network Virtualization - Operations

• Overall Logical network

health/stats

• VM to VM connectivity

• Per VM flow visibility

• Traffic Analysis – Packet

Capture

• Transport/Tunnel health

• Inventory/Fault Mgmt

• Multi-level Logging, Event

tracking and Auditing

• Physical network

troubleshooting/visibility

• Upgrade Management

Cloud Ops or Network Ops

vSwitch

NSX Edge ESXi ESXi ESXi ESXi

vSwitch vSwitch vSwitch vSwitch

L2

Logical Topology

L2

WAN/Internet

What are the key capabilities required for

operating a Logical world?

Controller Cluster

NSX Manager

Page 50: VMworld 2013: Operational Best Practices for NSX in VMware Environments

50 50

NSX Operations – Capabilities

NSX Optimized for vSphere

Logical Network Health UI: NSX Manager

CLI: Central NSX Controller, NSX Edge

VM to VM connectivity (Logical) NSX Controller Central CLI, Host level CLI

Traffic Flow visibility IPFIX (VDS)

NSX Edge – Flow Monitoring

Traffic Analysis per VM RSPAN/ERSPAN (VM Traffic)

Host Packet Capture (Overlay)

Network Inventory, Fault Management NSX Manager, SNMP (MIBS for ports, Switch etc)

Multi-level logging, Event tracking &

Auditing

Syslog Export (NSX controller, NSX Manager, NSX

Edge etc.)

Transport (Overlay) Health NSX Manager Connectivity Check

NSX Controller Central CLI, Per host CLI

Upgrade Management NSX Manager (Automated VIB and Controller

upgrades)

API visibility NSX Manager API

External Tools Custom, VCOPs, Log Insight

Page 51: VMworld 2013: Operational Best Practices for NSX in VMware Environments

51 51

NSX System Architecture

Cloud Management System

vCenter Server NSX Manager

Management Plane

Control Plane

NSX Edge

Logical

Router NSX Controller

Data Plane

NSX Edge

Services

Gateway

VXLAN DR DFW Security VXLAN DR DFW Security VXLAN DR DFW Security

vSphere API

Me

ssag

e B

us

vS

ph

ere

A

PI

REST API

NSX vSwitch

Control Plane Protocol

Control Plane

Protocol

REST API vSphere API

UWA

CP Protocol

Page 52: VMworld 2013: Operational Best Practices for NSX in VMware Environments

52 52

Control Plane Protocol

Control plane protocol

• All messages are TLVs

• Categorized into primitives:

• Connection management, negotiation, etc:

• Hello, bye, keepalive

• App specific:

• Open, close, notification, update, query

• Extensible.

• App ID in message common header.

VXLAN sub protocol • Update and query messages contains one

or more TLVs for different data types:

VM IP, VM MAC, VTEP

Page 53: VMworld 2013: Operational Best Practices for NSX in VMware Environments

53 53

VXLAN Control Plane Security

UW

Agent VTEP

UW

Agent VTEP

UW

Agent VTEP

vSphere Cluster B

UW

Agent VTEP

UW

Agent VTEP

UW

Agent VTEP

vSphere Cluster A

Controller Cluster

NSX Manager DB

1 Certificate

Generation

2 OVF

Deployment 3 Message Bus

4 REST API

NSX Manager

5 SSL 5 SSL

5 SSL

Page 54: VMworld 2013: Operational Best Practices for NSX in VMware Environments

54 54

Operational Practices – NSX Controller MAC Report

vSphere Host

VM

vSphere Distributed Switch

MAC1

Management

Network

10.20.10.10

vSphere Host

VM

MAC2

vSphere Host

Send VNI,VM

MAC Mapping

and VTEP IP to

Controller

1

2

5 VXLAN 5001

3

IP1 IP

2

Controller

VNI VM MAC

5001 MAC1

VNI VM MAC

5001 MAC2

10.20.10.11 10.20.10.12

6 7

4 8

VNI VM MAC VTEP

5001 MAC1 10.20.10.10

VNI VM MAC VTEP

5001 MAC1 10.20.10.1

0

5001 MAC2 10.20.10.1

1

Page 55: VMworld 2013: Operational Best Practices for NSX in VMware Environments

55 55

Operational Tools – NSX Controller IP Report

vSphere Host

VM

vSphere Distributed Switch

MAC1

Management

Network

10.20.10.10

vSphere Host

VM

MAC2

vSphere Host

Send VM MAC,

IP Mapping and

VNI to Controller

1

2

5 VXLAN 5001

3

IP1 IP

2

Controller

VNI VM

IP

VM

MAC

5001 IP1 MAC1

VNI VM

IP

VM

MAC

5001 IP2 MAC2

10.20.10.11 10.20.10.12

6 7

4 8

VNI VM IP VM MAC

5001 IP1 MAC1

VNI VM IP VM MAC

5001 IP1 MAC1

5001 IP2 MAC2

Page 56: VMworld 2013: Operational Best Practices for NSX in VMware Environments

56 56

Controller Based VXLAN – ARP Request

vSphere Host

VM

vSphere Distributed Switch

MAC1

Management

Network

10.20.10.10

vSphere Host

VNI VM

IP

VM

MAC VTEP

5001 IP1 MAC

1

10.20.10.1

0

VM

MAC2

vSphere Host

1

2

VXLAN 5001

3

IP1 IP2

Controller

10.20.10.11 10.20.10.12

6 5

VNI VM

IP

VM

MAC VTEP

5001 IP1 MAC1 10.20.10.10

5001 IP2 MAC2 10.20.10.11

4

Payload L2 DA: Broadcast

SA: MAC1

ARP Request for

VM IP2 sent to

Controller

ARP Report for

VM IP2, MAC2

sent to VTEP

10.20.10.10

VNI VM

IP

VM

MAC VTEP

5001 IP1 MAC1 10.20.10.10

5001 IP2 MAC2 10.20.10.11

Page 57: VMworld 2013: Operational Best Practices for NSX in VMware Environments

57 57

Controller Based VXLAN – Communication after ARP Resolution

vSphere Host

VM

vSphere Distributed Switch

MAC1

VXLAN Transport

Network

10.20.10.10

vSphere Host

VM

MAC2

vSphere Host

7

8

VXLAN 5001

IP1 IP2

Controller

10.20.10.11 10.20.10.12

9

VNI

V

M

IP

VM

MAC VTEP

5001 IP1 MAC1 10.20.10.10

5001 IP2 MAC2 10.20.10.11

Payload L2 DA: MAC2

SA: MAC1

L2 IP UDP VXLAN Payload L2

DA: 10.20.10.11

SA: 10.20.10.10 5001

10

DA: MAC1

SA: MAC2

VNI VM

IP

VM

MAC VTEP

500

1 IP1 MAC1 10.20.10.10

500

1 IP2 MAC2 10.20.10.11

VNI VM

IP

VM

MAC VTEP

500

1 IP2 MAC2 10.20.10.11

VNI VM

IP

VM

MAC VTEP

500

1 IP2 MAC1 10.20.10.11

500

1 IP1 MAC2 10.20.10.10

Page 58: VMworld 2013: Operational Best Practices for NSX in VMware Environments

58 58

Operational Tools – NSX Edge Services

Download Edge Gateway Tech Support Logs using the Web Client

Or from NSX Edge CLI using the following command

NSX-Edge1-0# export tech-support scp user@scpserver:file

Page 59: VMworld 2013: Operational Best Practices for NSX in VMware Environments

59 59

Operational Tools – REST API

• VERB = GET

• URI = https://<NSX Manager Hostname>/api/2.0/vdn/scopes

• HEADERS = Authorization

• HTTP Body = N/A

• RESPONSE: Search for the id of scope:

<id>vdnscope-X</id>

Page 60: VMworld 2013: Operational Best Practices for NSX in VMware Environments

60 60

Operational Tools – REST API

• VERB = POST

• URI = https://<NSX Manager

Hostname>/api/2.0/vdn/scopes/vdnscope-1/virtualwires

• HEADERS = Authorization, Content-Type

• HTTP Body =

<virtualWireCreateSpec>

<name>Test-Logical-Switch-01</name>

<description>Created via REST API</description>

<tenantId>virtual wire tenant</tenantId>

<multicastProxy>true</multicastProxy>

<disableMulticast>true</disableMulticast>

</virtualWireCreateSpec>

Page 61: VMworld 2013: Operational Best Practices for NSX in VMware Environments

61 61

Operational Tools – REST API

• Response: 201 Created

• The Response Body provides the virtualwire-id, which can be used for additional

operations (eg, attaching to a Logical Router LIF or for Distributed Firewall rules)

Page 62: VMworld 2013: Operational Best Practices for NSX in VMware Environments

62 62

Demo 1 (3 mins) - Script

Component Installation

• NSX Manager

• NSX Controller Cluster

Preparation

• Login to ESXi host (destination)

• Add Logical Switch

• Connect VMs to Logical Switch

Data Collection

• Start data collection on destination host, output to a share (that is also accessible on analyzer)

• Connect VMs to Logical Switch

• Generate some traffic

• Stop data collection

• Start Wireshark and open pcap file

• Enable VXLAN decoder

• Walk through packet data format (VXLAN headers, unicast mode etc)

• Show anything else ? Controller CLI/esxcli ?

Page 63: VMworld 2013: Operational Best Practices for NSX in VMware Environments

63 63

References

Other VMworld breakouts – VXLAN troubleshooting,

Security operations

VMware Networking and Security Booth

Hands on Lab: HOL-SDC-1303 VMware NSX to gain hands

on experience

Expert Bar/Group Discussions

Page 64: VMworld 2013: Operational Best Practices for NSX in VMware Environments

64 64

Other VMware Activities Related to This Session

HOL:

HOL-SDC-1303

VMware NSX Network Virtualization Platform

NET5790

Page 65: VMworld 2013: Operational Best Practices for NSX in VMware Environments

THANK YOU

Page 66: VMworld 2013: Operational Best Practices for NSX in VMware Environments
Page 67: VMworld 2013: Operational Best Practices for NSX in VMware Environments

Operational Best Practices for NSX in VMware

Environments

Ray Budavari, VMware

Thomas Kraus, VMware

NET5790

#NET5790