Deploying Efficient OpenStack Clouds

Yaron Haviv

yhaviv@gmail.com

Agenda

• OpenStack Overview, Benefits, and Challenges

• Overall topology and setup

• OpenStack Networking

• Options, Overlays, Challenges

• Storage in OpenStack

• Options (Ceph, iSCSI, iSER, Swift, HDD/SSD), Pro and Cons

• Building Cost effective solutions with application in mind

• Summary

• Based on commodity

• Reduce software costs

• Automation

What Does OpenStack Buy You ?

Flexibility & FeaturesCost Effective Cloud

Integration & Interoperability

• Endless amount of services

• OpenSource and Extensible

• Many configuration and scripting options

• Compute, Storage, Networks, and Apps under one system

• Modular approach with Pluggable services

• Testing, integration, and certification by multiple vendors and communities

Deploy OpenStack, Practical View ?

• Require strong IT group with DevOps professionals, not suitable for small/non-technical organizations (yet)

• Must think cloud to take advantage, and not tie your self to Enterprise paradigms

• Start with solid distribution (RedHat RHOS, Ubuntu, Mirantis, ..) or use professional services to ramp up

• It’s a community, if you want to get help, you must also contribute

• Test on small scale, validate assumptions, before going to production setups

Typical OpenStack Deployment

Firewall, L3 Routing, NAT, DHCP services

(can have multiple)

CinderiSCSI or Ceph based storage

NovaCompute (VM) nodes

Management servicescan also be installed on

the same server

Source: stackops.org

OpenStack in Converged Network Environment

Native integration of

Mellanox products with

Neutron

Hardware-based support for

security, and isolation functions

Accelerating storage

access by up to 5x

DPDK DPDK

Source: Mellanox

Types of Networks in OpenStack

• Hypervisor Networks

• Console

• Storage

• Messaging

• VM (Tennant) Networks

• VLAN Based

• Overlay based (VXLAN, GRE)

• Connection/Port Types

• Para-virtualized, e.g. using OpenvSwitch

• Direct Attached, using SR-IOV, for highest performance & native NIC features (RDMA, OS Bypass, ..)

Logical View of an OpenStack Multi-Tenant Cloud

Implemented via software or hardware appliances

Using VLANs or Tunnels (VXLAN)

Mellanox Network Virtualization (Neutron) Plug-in

OSVM

Para-virtual

OSVM

OSVM

OSVM

tap tap SR-IOV to the VM

Provision VM & Fabric Policy in hardware, through standard APIsBenefits: Isolation, functionality, performance & offload, simpler SDN

Embedded Switch

Mellanox Nova

Plug-in

Create/delete, configure policy per VM vNIC

Neutron Plug-Ins

Servers

Manager

OpenStack Manager

Compare eSwitch vs OVS Qperf (TCP) Latency

Source: Mellanox

Typical VxLAN Overlay Network Deployment

OSVM

OSVM

OSVM

UDP

OSVM

BR0 BR1

VXLAN Overlay

IP

VNI100 VNI300Open vSwitch (OVS)

Underlay Network (Layer 2 or Layer 3)

Hypervisor

vTap vTap vTap vTap

OSVM

OSVM

OSVM

UDP

OSVM

BR0 BR1 BR2

VXLAN Overlay

IP

VNI100 VNI200 VNI300Open vSwitch (OVS)

Hypervisor

vTap vTap vTap vTap vTap

VxLAN Overlay (tenant) networks

UDP

BR0 BR1

VXLAN

IP

VNI200 VNI300

Router / NAT Software Gateway

vEth1vEth0

Internet

Gateway (Network) Nodes

OVS

SDN Manager e.g. OpenStack Neutron, ODL,

VMware NSX

What is VxLAN

• “Virtual Extensible LAN (VXLAN) is a network virtualization technology that attempts to ameliorate the scalability problems associated with large cloud computing deployments. It uses a VLAN-like encapsulation technique to encapsulate MAC-based OSI layer 2 Ethernet frames within layer 3 UDP packets.” Wikipedia

Performance & CPU Advantage Using VxLAN Offload

1 VM 2 VMs 3 VMs

NO VxLAN

11 19 21

VxLAN in software

2 3 3.5

VxLAN HW Off-load

10 19 21

2.512.522.5

Total VM Bandwidth when using VxLANB

and

wid

th [

Gb

it/s

ec]

1 VM 2 VMs 3 VMs

NO VxLAN

0.545454545454545

0.68421052631579

0.666666666666667

VxLAN in soft-ware

3.5 3.33333333333333

4.28571428571429

VxLAN HW Off-load

0.9 0.894736842105263

1.19047619047619

0.50

2.50

CPU Usage Per Gbit/sec with VxLAN

CP

U%

/ B

an

dw

idth

(G

bit

/sce

)

5x More Bandwidth with VxLAN Offload4x Less CPU

Overhead with VxLAN Offload

Test Details- Test command: netperf -t TCP_STREAM –H- 1-3 VMs talking to 1-3 VMs on a second server- OpenvSwitch (OVS) with VxLAN support- Servers:

- HP ProLiant DL380p Gen8- 2 x Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz- 32GB RAM

- Hypervisor OS: Linux Upstream Kernel 3.14-rc1 + KVM- Guest VM OS: RHEL 6.5 2.6.32-431.el6.x86_64 - NIC: ConnectX-3Pro , FW: 2.30.8000- CPU% and Bandwidth measures on the Hypervisor (aggregate

1-3 VMs)

Source: Mellanox

Large Scale Topology Diagram

Example: Redundant and shared storage connectivity, and non-redundant server connectivity (to save costs)

OpenStack Storage OptionsOption Pro Con Perf $/GB*

Ceph • Free • Multiple front-ends• Built in HA, Snaps,

DR, ..

• Slow & CPU intensive

• Complexity & Stability

• Replication (with apps like Hadoop)

Low-Med

0.40

iSCSI-LVM or iSER-LVM

• Free• Fast

• HA Features • Cinder only

Med-High

0.20

Commercial • Performance• Stability & Support• Features

• Cost• Flexibility

Med-High

0.50 - 2

Swift • Free• Low-cost

• Performance • Object only

Low 0.30

Reference: Amazon AWS

• No OpEx overhead • Slow• data transfer

costs

Low 0.30/yr

* - all inclusive, using 4TB HDDs, with redundancy

Ceph (Vision) – All Purpose Storage

Cinder iSCSI Deployment Example

iSCSI or iSER Storage Servers/Heads

JBODs or Mid-range RAID Enclosure

HA

SAS/SATA

OpenSource software (Cinder LVM) or commercial software/system

Initiators/Hosts accessing through iSCSI or iSER

(iSCSI RDMA)

Note: need to use the right ratio of disks per Head to optimize cost/performance

With 40GbE and RDMA offload, each head can support 5x more disks, lowering overall costs

• Using OpenStack Built-in components and management (Open-iSCSI, tgt target, Cinder), no additional software is required, RDMA is already inbox and used by our OpenStack customers !

RDMA Provide Fastest OpenStack Storage Access

Hypervisor (KVM)

OSVM

OSVM

OSVM

Adapter

Open-iSCSI w iSER

Compute Servers

Switching Fabric

iSCSI/iSER Target (tgt)

Adapter Local Disks

RDMA Cache

Storage Servers

OpenStack (Cinder)

Using RDMA to accelerate

iSCSI storage

1 10 100 10000

1000

2000

3000

4000

5000

6000

7000iSER 4 VMs WriteiSER 8 VMs WriteiSER 16 VMs WriteiSCSI Write 8 vmsiSCSI Write 16 VMs

I/O Size [KB]

Band

wid

th [M

B/s]

PCIe Limit

6X

RDMA enable 6x More Bandwidth, 5x lower I/O latency, and lower CPU%

Source: Mellanox

Storage TiersWorkload Comment $/GB

(Raw)

HDD Big Data workloads (e.g. Hadoop, Video)

Very slow IOPs, VM cross interference

0.04

HDD with SSD as Cache(e.g. LSI FlashCache, Bcache, commercial)

Localized disk access (e.g. VM Images)

0.20

SSD Storage IOPs/latency sensitive apps (e.g. Databases, VDI)

1-2

SSD/TLC Read mostly, Random Fast reads Low-endurance SSDs

0.40

Challenge: How to pre-allocate storage to different tiers and different usage models (Block, Object, Files, ..)

Food For Thought

Should I use 4TB or 2TB Disks ?

$/GB cost with 60 x 3.5” HDD JBOD(Disks + Enclosure costs / Disks #)

Disk $/GB

4TB 0.08

2TB 0.11

For 30% extra cost you can double the IOPs & BW

Highly Efficient Rack Example U36 27 2835 25 2634 23 2433 21 2232 19 2031 17 1830 15 1629 13 1428 11 1227 9 1026 7 825 5 624 3 423 1 2222120 Storage19

18 Storage17

16 Storage15

14 Storage1312 Storage1110 Storage98 Storage76 Storage54 Storage32 Storage1

Switch: 36 x 40/56Gb (IB/Eth)2nd Switch (Optional)

28 x 0.5U (twin) servers• Each with Mellanox ConnectX3-Pro 40/56Gb

Adapter• Optional up to 6 Disks per server

2 x High performance storage servers, each with • 2U 24 bay server + 2 x 45 bay JBODs • 24 x 2.5” 500-1000GB SSDs • 90 x 3.5” 2-4TB SAS/SATA HDDs• 1-2 x ConnectX3-Pro 40/56Gb Adapter• Run Cinder/LVM (iSCSI/iSER) and/or Ceph

1-2 x Switches (e.g. Mellanox SX1036/6036)• 28 x 40/56Gb copper links to servers • 4 x 56Gb Copper links to storage • 4 x 56Gb optical uplinks to core

Thank You !Yaron Haviv, yhaviv@gmail.com

Manila – File Shares as A Service

What is Manila?– Multitenant, secure file share as a service– “Cinder for shared file systems”– NFS & CIFS protocols supported today; more to come

Will be available Junos

OpenStack Netron Network Topology Examples