OpenStack Load Balancer as a Service (LBaaS) with openSUSE Leap
ScalableOpenStackNetworkswithNSX … inmixedKVM/ ESXi Environments ... OpenStack’ Networking ......
62
Scalable OpenStack Networks with NSX in mixed KVM/ESXi Environments Tom Schwaller Senior Systems Engineer (NSX, VIO, CNA) | VCIXNV EMEA SDDC PreSales [email protected] | @tom_schwaller
Transcript of ScalableOpenStackNetworkswithNSX … inmixedKVM/ ESXi Environments ... OpenStack’ Networking ......
Scalable OpenStack Networks with NSXin mixed KVM/ESXi Environments
Tom SchwallerSenior Systems Engineer (NSX, VIO, CNA) | VCIX-NVEMEA SDDC Pre-[email protected] | @tom_schwaller
Agenda
§ VMware Integrated OpenStack (VIO)§ Overview & Architecture§ VIO 2.5 - New Features§ Simple VIO Deployment§ NSX-v Integration
§ NSX-T and OpenStack
VMware Integrated OpenStack -Overview
What is VMware Integrated OpenStack (VIO)
vSphere NSX vSphere Datastores(SAN/NAS/VSAN)
Standard OpenStack
Nova Neutron Cinder
Keystone HeatHorizon Ceilometer
Glance
VIO Management Server
(Deploy, configure, patch, upgrade OpenStack)
VIO vRealize Operations
vRealize Log Insight
vRealize Business
• VIO is an “Integrated Product” Approach to OpenStack• Standard OpenStack Distribution (delivered as OVA)• Runs on Top of VMware SDDC• Fully supported by VMware (no License Cost, optional Support $200/CPU/year)
Deploy Patch Upgrade Monitor Troubleshoot
VIO vCenter Integration
Troubleshooting with vRealize Operations Manager• NSX Management Pack for vROPS
• OpenStack Management Pack for vROPS.
Troubleshooting with vRealize Log Insight• NSX Content Pack for Log Insight.
• OpenStack Content Pack for Log Insight (planned).
2010
Open vSwitch Project created by Nicira
2011
OpenStack Project created by Rackspace and NASA
VMware joins OpenStack Foundation as Gold member
2013
Nova vCenter Driver
Cinder VMDK Driver
Neutron NSX DriverCeilometer vCenterDriver
Leading OpenStack Distros support vSphere & NSX
VIO Beta AnnouncedGlance VMDK Driver
OpenStack distribution partners announce plans to support vSphere and NSX
2014
OpenStack Networking “Neutron” project started, led by Nicira
2012
NSX+Neutron
Long-Standing Commitment to OpenStack Community
VIO 1.0 GA• Icehouse Based
• Deploy Production OpenStack in Minutes
VIO 2.0 GA• Kilo Based
• First Kilo DefCoreCompliant Distro
• Seamless, Non-Disruptive OpenStackUpgrade
VMware Integrated OpenStack Launches. Strong Momentum with Customers
2015
VMware’s Community Involvement By the Numbers
Source: Stackalytics for tc-approved OpenStack projects in the OpenStack Mitaka release (http://stackalytics.com/?release=mitaka&project_type=tc-approved-release&company=vmware)
Top 10 contributor to the OpenStack releases
20 Developers 189 Commits
14690 LinesOf Code
1553 Patches Reviewed
Some VMware Integrated OpenStack Customer ReferencesCustomer 1 Customer 2 Customer 3
Deployment Size
Workload Profile
VIO Benefits
5000+ VMs
E-Commerce WebSite
• 10 Weeks to Production
• 4 Existing Employees Running Entire OpenStack Cloud
• Leverage vSphere HA & vMotion to protect workloads
• Use VIO built-in automated patching to address issues
800+ VMs
Analytics
• < 8 weeks to Production
• Leverage vSphere Clustering to replace server & storage: Zero Downtime to OpenStack Cloud
• Leveraging NSX for complete multi-tenant L2-L7 networking
1000+ VMs
CI/CD Pipeline
• 1 FTE running the cloud
• Upgraded OpenStack from Icehouse (VIO 1.0) to Kilo (2.0) all by themselves!
• Leveraging vSphere to reliably run Windows VM
Completely Supported by VMware
VIO Architecture
Horizon(web portal) CLI Tools / SDKs
vRBCost visibility, governance, etc...
vCenter
Nova(compute)
Neutron(network)
Cinder(block storage)
Glance(images)
vCenter Datastores3rd-party / VSAN
Keystone(identity)
Local DB LDAP
Log Insight Log collection, O/S Content pack
NSX
vROPsOpenStack mgmt pack
Included OpenStack Components:Integrated VMware Technologies:
Current
Swift(object store)
Basic open source
vSphere Install, Configure and Troubleshoot
3rd Party Object Storage
Heat CeilometerHeat Auto Scaling
VMware Integrated OpenStack 2.0 - Components
VIO 2.0 Architecture
Users (+ cloud brokers, etc.)
External network (API Access)
Load Balancers (HA pair)
Public Virtual IP
Private Virtual IP
management network
OS Controller 2
OS Controller 1
OS API / Horizon Nodes (active/active)
OpenS
tackcore
vSphere
/ VIO mgmt
NSX Manager
vCenter / SSO
VIO Manager
memcache 1
memcache 2
RabbitMQ 1
RabbitMQ 2
(active/active)
(active/active)
OS mpute 2
Nova Compute 1N times
Nova Compute (per vSphere Cluster)
DB 1
DB 2
(active/active DB Cluster)
DB 3
vSphereCluster(s)
vSphereDatastores
vSphereDatastores
vSphere Datastoresfor Glance (Images)
VIO Architecture - Networks
NSX Manager
vCenter/SSO
VIO Manager
VIO 2.0 - Management Cluster
OpenStack Storage Types on VMware
• Nova • Cinder • Swift • Glance
• Root Disk
• Non-persistent
• Volumes
• Persistent
• Blob Storage
• Rest APIs
• Catalog
• Object/Block Backend
Partner SolutionvCenter Driver VMDK Driver VMDK Driver
Ephemeral Block Object Image
vCenter
Cinder VMDK Driver• Multiple volumes on single Datastore• Multiple disk formats– Thin, thick, thick eager zeroed
• Leverage vSphere Features– S-DRS– S-vMotion (broken)– Policy Based Storage Mgmt (PBSM)
• Backend & Protocol agnostic– iSCSI, Fiber Channel, FCoE, NFS, VSAN– Future: vVols
SAN / NASx86 Servers
Virtual SAN
NFS Server
Cinder
VMDK Driver
iSCSI / FCoE / FC NFS Mount
Attach Volume
How Cinder VMDK Driver Works
VMware vCenter Server
VMware ESXi
Nova CinderCinder executes Volume Operations through VMDK Driver
VMware ESXi
Glance
VMFS/vSAN
Create Volume
vCenter creates the volume and initially volume belong to a Shadow VM (never turned on)
Cinder VMDK Driver Preserves all vSphere rich Storage Features: SDRS, S-vMotion, VAAI Acceleration,…
VMDK Driver
Shadow VM
When volume is attached to VM, vCenter changes the parent for the volume
Cinder & SPBM• Create vSphere storage policies• Create Cinder volume types• Create extra spec– Volume Type è Storage Policy
• Create Volume with type specified• Benefits– Storage Tiering– Storage QoS
vCenter
SAN / NASVirtual SAN NFS Server
CinderVMDK Driver
Gold Policy Silver Policy Bronze Policy
Gold Vol Type Silver Vol Type Bronze Vol Type
VMware Integrated OpenStack 2.0 - Features
KiloSeamless
Upgrade from VIO 1.0 with rollback
6 Additional Languages (German, French,
Chinese (Traditional/Simplified), Japanese, Korean)
Multi Region,Multi Hypervisor
Support(with another
OpenStack Distro)
Now Included in Federation Enterprise
Hybrid Cloud 3.5
Ceilometer Support & Heat Auto Scaling
LBaaS Support (Load Balancing as a
Service)
Qcow2 Image Format Support
Backup & Restore for OpenStackServices and Configuration
Advanced Workload
Placement using vSphere Affinity / Anti Affinity
VIO 2.5 - New Features
VIO 2.5: Feature List§ Compact 7 VM Architecture with a highly available Control Plane§ Single VM VIO (Tech Preview)§ Import vSphere Templates into OpenStack§ NSX-T Support (Tech Preview)§ Built in basic Monitoring Tool to get Health & Status of VIO Deployment using viocli§ OpenStack API Profiling§ Tool to detect and fix DB Sync Issues§ Cross Cluster Live Migration of Nova Instances§ Cross Datastore Migration of Cinder Volumes and Nova Instances§ Streamlined Glance Image handling for improved Performance and Reliability of Instance Boot and Snapshots
§ Neutron L2 Gateway (Overlay Network to VLAN Bridging)§ Capacity Subscription to guarantee Resource Allocation for Tenants§ SR-IOV
VIO 2.5 ArchitectureUsers (+ cloud brokers, etc.)
External network (API Access)
Load Balancers (HA pair)
Public Virtual IP
Private Virtual IP
management network
Controller 2Memcache 2
Controller 1Memcache 1
API / Horizon Nodes (active/active)
OpenS
tackcore
vSphere
/ VIO mgmt
NSX Manager
VIO Manager
OS mpute 2
Nova Compute 1
N times Nova Compute
(per vSphere Cluster)
DB 1 / RabbitMQ 1
DB 2 / RabbitMQ 2
(active/standby DB Cluster active/active MQ cluster
DB 3 / RabbitMQ 3
vSphereCluster(s)
vSphereDatastores
vSphereDatastores
vSphere Datastoresfor Glance (Images)
viocli deployment -d >deployment_name> getlogs
• Reduced management cluster footprint
• Full HA: No Service Downtime
• Database replication: No Data Loss
• 6000+ VMs
VIO 2.5 - Single VM VIO• For VIO 2.5 Demos/PoCs• Deploy VIO OVA Image• Edit /opt/vmware/vio/etc/omjs.properties on OMS
oms.deployment_type=singlevmoms.disable_hosts_anti_affinity = true
oms.skip_cluster_vmotion_check = trueoms.singlevm.cpu.size=4
oms.singlevm.mem.size=8192
• Restart OMS service: service oms restart• Deploy VIO with the vSphere Web Client
VIO 2.5: Import vSphere Templates as Glance Images
• Seed VIO with existing VM templates• Start deriving value from VIO quickly
vCenter
OpenStack
Glance Image
Glance Image
Glance Image
Glance Image
VM Template
VM Template
VM Template
VM Template
glance image-create --name my_first_template --disk-format vmdk --container-format bare --location ”vi://<vc_hostname>/DC_Folder/Datacenter1/vm/MyTemplates/ubuntu14.04”
• New CLI Command in VIO 2.5viocli deployment status
• Reports the following Problems: • Time out of sync among Management Server and any OpenStack Nodes.• Any missing vital OpenStack and dependent Processes.• Number of active Members of OpenStack Database Cluster.• Number of OpenStack Database per OpenStack Service, e.g. Nova• Broken Network Connections among services. (Experimental)
VIO 2.5: Basic Monitoring Tool
• Quick snapshot of VIO health• Troubleshoot Failures
VIO 2.5: Basic Monitoring Tool• Includes Log Insight Dashboard
• Quick view of failures
VIO 2.5: DB Sync Tool - Overview
• Detect inconsistencies between OpenStack & vSphere/NSX state
• Fix sync issues
vSphere NSX
Nova
vCenter Driver
Cinder
VMDK Driver
Glance
VMDK Driver
Neutron
NSX Driver
OpenStack
Management
Server
VM
Volume
Image
Network
DHCP
Router
SecGroup
Sync State
VIO 2.5: DB Sync Tool - viocli inventory-admin
• New CLI command in 2.5• Compares Nova/Cinder Inventory with vSphere Inventory• Reports orphaned Objects (in 2.5 Instances, Instance VMs, Shadow VMs for Volumes)
• Example: Show orphaned Instances (that exists in Nova, but do not exist in vSphere).
% viocli inventory-‐admin show-‐instances
+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+| tenant_name | id | name | created |+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+| alt-‐tenant | a2dcc20f-‐b10c-‐4a53-‐bb85-‐0d8c16d13952 | test | 2016-‐02-‐06T01:29:43Z |+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+
VIO 2.5: Cross Cluster Migration• Support for Nova Live Migrate API
• Migrate instances across vSphere Clusters
ESXi
ESXi
ESXi
Cluster 1
ESXi
ESXi
ESXi
Cluster 2ESXi
ESXi
ESXi
Cluster 3vCenter
Nova
Compute Host 1
Compute Host 2
Compute Host 3
Migrate
VIO 2.5: Cross Datastore Migration• Migrate volumes across datastoresCinder
VMDK Driver
vCenter
Datastore 1 Datastore 2
- Cinder Volume
1
Migrate prep (1 Command)
2
Datastore Maintenance Mode / S-DRS / S-vMotion
3
Migration transparent to Cinder
Migrate
L2 Gateway - User Scenario & Solution • Customer needs VM on VLAN. But in a L3 Leaf/Spine fabric, the Compute-Cluster doesn't have access to that VLAN.
• Solution
– VXLAN Network + L2 VXLAN/VLAN bridging
On controller01 (CLI only: Use neutron-l2gw Client)# neutron-l2gw l2-gateway-create L2GW1 --device name=<dummy name >,interface_names=<port group moid>
# neutron-l2gw l2-gateway-connection-create <l2 gateway name> <tenant network name> --default-segmentation-id <vlan id>
Simple VIO Deployment
VIO Networking -NSX-v Integration
VIO: vSphere and NSX Interaction - High Level
Nova Compute Nova Services
NSX Manager
Neutron Plugin
Neutron Server Heat
Glance RabbitMQ
vCenter
ESXi-2
vSphere Plugin
NSX
VIO
ESXi-1
NSX
KeystoneCinder
VMDK Driver
VMDK Driver
Networks and Security Services available in VIO/NSX• Full Neutron feature-set– Private Logical Network Identifier Independent of VLANs– DHCP Service– Security Groups – Metadata Service Integration & Support– L3 (centralized, distributed) and static route– NAT & Floating IP– LBaaS support (VIO 2.0)
• Enterprise Features– Micro-Segmentation with line-rate stateful Distributed Direwall– In-Kernel Distributed Routing– No-NAT Support– Provider-side security via Service Insertion– Support for multiple vDS for Edge & Mgmt clusters to enable network design flexibility (VIO 2.0).
NSX vSphere Neutron Plugin - DHCP Implementation
VLAN/VXLAN Trunk
DHCP Server(NSX ESG) DHCP Servers
(NSX ESG)
Non-overlapping IPs Overlapping IPs
User A
User B
User C
User A
User B
User C
172.16.10.0/24
172.16.20.0/24
172.16.30.0/24
172.16.10.0/24
172.16.10.0/24
172.16.10.0/24
• Single Edge used to aggregate multiple networks requiring DHCP
• VLAN/VXLAN trunks (sub-interfaces)• 200 sub-interfaces per Edge
• Individual Edge per overlapping IP subnet
NSX vSphere Neutron Plugin - NSX Edges Pool
Backup NSX ESGRouters and DHCP Servers
Pre-populated Edge PoolBackup NSX
Distributed Routers
• In order to improve usability, the Edge Cluster must be “pre-heated” with pre-provisioned Edges• Edges serve multiple purposes: centralized routing, distributed routing, load balancers and DHCP• Provisioning in OpenStack = Edge reconfiguration in NSX = better response time
NSX vSphere Neutron Plugin - Supported Topologies
• VLAN or VXLAN tenant networks• NAT, no-NAT NSX Edge Gateways• LBaaS• Static routing only
Multi-tier Application with Centralized Routing Services
Multi-tier Application with Centralized and Distributed Routing Services
• VXLAN-only tenant networks• NAT, no-NAT NSX Edge Gateways• LBaaS• Static routing only
NSX vSphere Neutron Plugin - Distributed Routing + NAT
NSX OpenStack Neutron
In NSX, a topology involving a distributed router and NAT services
requires two routers
In Neutron, both the distributed router and the centralized router will be
represented as a single Neutron router
NSX vSphere Neutron Plugin - A word on Dynamic Routing…
Transit Logical Switch
User
VM1 VM2
Logical Switch A
VM5
Logical Switch B
Distributed Logical Router
Router(NSX ESG)
VM3
VM4
Physical Routers BGP/OSPF
StaticRouting
Provider Router (NSX ESG HA)
StaticRouting
Provider
• No dynamic routing support in Neutron (future enhancement)
• Dynamic routing can be enabled between provider Routers and physical routers
Load Balancing as a Service - LBaaS
User
Web1 Web2
Network A
VM5
Network B
Load Balancer(NSX ESG)
Web3
VM4
• VIO 2.0 supports LBaaS 1.0• Synchronous API Calls• No SSL Termination Support
• VIO 2.5 supports LBaaS 2.0• Inline only (with Dedicated ESG)
VIP
TCP/HTTP/HTTPS
NSX vSphere Neutron Plugin - Metadata Services (1)• The VIO Neutron Plugin for NSX vSphere will automatically provision a pair of NSX ESGs for Metadata routing.
• These Metadata ESGs connect to a Logical Switch that Tenant Routers will also connect to, on a /17 subnet.
• On the external side, the Metadata ESGs are linked to the Management network, where the rest of the VIO infrastructure sits, including of course the Metadata Service (VIO Controllers). The NSX ESG Firewall ensures that only metadata traffic flows between the instance and the Nova Metadata service.
Metadata Routers (NSX ESG)
Metadata Service(VIO Controllers)
Tenant Routers(NSX ESG)
Shared Logical Switch169.x.x.x/17
VIO Management Network (default)
orVLAN/PortGroup**
NSX vSphere Neutron Plugin - Metadata Services (2)• For instances sitting on networks without a Neutron router, the DHCP ESG is responsible for connecting the instances to the Metadata service.
Metadata Routers (NSX ESG)
Metadata Service(VIO Controllers)
DHCP Edge(NSX ESG)
Shared Logical Switch169.x.x.x/17
Tenant NetworkVM1 VM2
VIO Management Network (default)
orVLAN/PortGroup**
NSX vSphere Neutron Plugin - Security Groups
Neutron Security Group rules are mapped to dedicated NSX Distributed Firewall rules, organized
into sections
NSX vSphere Neutron Plugin - SpoofGuard
Neutron networks are mapped to individual NSX SpoofGuard policies,
sharing the same UUID
MAC to IP association allows NSX to leverage objects in the DFW Security
Groups
NSX-T and OpenStack
NSX-T• It is completely new (no official VMware Announcement)• Decoupled from vCenter - Host Switches are NSX objects but no VM Visibility• API driven (developer focused), Objects are based on UUID• Good Workflows and Troubleshooting Tools
56
vCenter
Management/Edge Cluster
Compute - ESX Hosts
Compute - KVM Hosts
MANAGEMENT
VTEP
ESX
VTEP
KVM
UPLIN
KONE
UPLIN
KTWO
NSX Managers NSX Controllers NSX EdgeNodes
WebVM
WebVM
AppVM
AppVM
DBVM
DBVM
WebVM
NSX-T: Network Layout Overview
Router
VIO & NSX-T Installation
• 10 VIO nodes are deployed
Switching - OpenStack Configuration• Create a Tenant Network(OpenStack Network) with Horizon UI– Under "Project - Network - Networks", Create Network
neutron net-create Web-Netneutron subnet-create --name Web-Subnet Web-Net 192.168.10.0/24
Switching - What happens in the Backend
• A logical switch is created during OpenStack create a network
– Login to NSX manger, click SWITCHING on the navigator
NSX-T Multi-Tier Logical Routers• NSX-T Multi-Tier routing architecture is new– NSX-T Tier0/Tier1 model doesn’t map to NSX-v Edge/ Distributed Logical Router
– Ideal Architecture for OpenStack Integration
• Provider Logical Router - Tier0 LR– Role: Attract and send ECMP Services– Manual Management– NSX-T Infrastructure Administrator
• Tenant Logical Router - Tier1 LR– Role: Per Tenant First Hop Router &stateful Services– OpenStack management
eBGPeBGPTier 0 Logical Router
Tier 1 Logical Routers
Physical Infrastructure
Thank You!
Where to Learn More
Hands-on-Lab• HOL-SDC-1620: VIO with vSphere and NSX• Online at: http://www.vmware.com/go/openstacklab
Have Questions?• Visit our online community: http://communities.vmware.com/community/vmtn/openstack
Download VIO + Learn Morehttp://www.vmware.com/products/openstack
OpenStack Training• Free 3-hour online training course on running OpenStack on VMware infrastructure
http://www.vmware.com/go/openstacktraining
Blog / Twitter • Read http://blogs.vmware.com/openstack/• Follow @VMware_OS
