OpenStack Architectural Decisions · – Total vCPUs = 2 x 8 x 2 = 32 vCPUs (Do take into account...

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

OpenStack Architectural Decisions How to architect OpenStack for your organization Thomas Goh


Agenda

• The General Purpose Cloud • General Considerations • Compute Considerations • Network Considerations • Storage Considerations


General Purpose Cloud

•  Good starting point • Balance components and features • Equal weight to

– Compute – Network – Storage

• Not specialized workload or edge case • 80% of potential use cases • Designed for future workloads


General Purpose Cloud

• Common Use Cases –  Providing a simple database –  A web application runtime environment –  A shared application development platform –  Lab test bed –  scale-out workloads


General Purpose Cloud Basics •  Virtual-machines •  Disk image library •  Block storage •  Object storage •  Firewalls •  Load balancers •  VPNs •  IP address Management •  Network overlays •  Software bundles


General Considerations


General Considerations

• Cost – Primary concern for any organization – Private vs Public Cloud Cost structures

• Enterprise vs Commodity • Margins for public clouds

– CapEx vs OpEx vs Lease/Loan

• Time to market – Cloud Deployment – Products and services delivery

• Agility • Self-Service • On-Demand • Elasticity


General Considerations • Performance

–  Majority of resources (Compute, storage, network) should target general workloads –  Provide targeted performance enhancements

• No predefined usage model –  Should be designed to handle 80% of the workloads

• On-demand and self-service application –  The self-service model allows users to scale out their resources, systems should be able to handle such

operations without disruption.

• Security –  Clouds requiring significant specialized security requirements goes against a general purpose cloud and will

significantly increase complexity.

• Scalability and Resource Planning –  Cloud systems are easy to scale –  Resource planning is important to ensure there is enough capacity to satisfy user demand


General Considerations • Public cloud

–  May examine commodity hardware and financial leasing to increase margins and ease capital expenditure

• Private cloud –  Is it logical to create their own clouds internally –  CapEx vs OpEx –  Increase control over technical architecture and capabilities.

•  Legal –  Data retention –  Data Ownership –  Data Sovereignty –  Data compliance


General Considerations • Large org with complex needs

– Need to identify appropriate workloads or use cases

• Legacy infrastructure – Legacy infrastructure not really suited for cloud – Special consideration needed for high-availability

• Corporate culture (Slow vs Fast) – Will need to establish a proper roadmap to the cloud for slow organizations

•  IT Competence – The base cloud infrastructure requires specialized knowledge from IT generalist. – Need to manage cloud in an agile manner – Must learn about automation and distributed technology


General Considerations • Stage in Cloud Journey

– Are you ready for the cloud? – Have you standardized, virtualized and automated yet?

•  Internally Managed – Do you have the technical competency to manage a cloud infrastructure and deploy legacy

and cloud native application

• Outsourced – Does your outsourced agency have the capabilities to manage an end-to-end cloud


Cloud Architecture Examples


What your cloud might look like

Hardware

OpenStack Operations

KVM

Redhat

Ceph

Automation

Monitoring

Log Mgmt

Provisioning OS

Swift

CPU

Cloud Applications Cloud Operations

Network Storage Compute

OpenStack API

HyperV VMWare Bare Metal

SAN


General Architecture


Compute Considerations


Compute Considerations •  General Considerations

–  Processor Type –  Memory –  Local Storage –  Operating Systems & Hypervisors

•  Compute Pools –  Multiple pools for specific workload types

•  Compute/Memory/Storage Intensive •  Instances are scheduled to appropriate pool

•  Consistent hardware design –  CPU Type (can cause issues migrating between hosts) –  Storage (Inefficient storage usage) –  Memory (Inefficient memory utilization)



• Overcommit ratio – OpenStack is able to configure the overcommit ratio for CPU and memory – Default CPU overcommit ratio is 16:1 and the default memory overcommit ratio is 1.5:1

(Personally I do not like to over commit) – Consider a m1.small instance uses 1 vCPU, 20 GB of ephemeral storage and 2,048 MB of

RAM • Assuming a server with 2 CPUs of 10 cores each, with hyper-threading • Default CPU overcommit ratio of 16:1 would allow for 640 (2 × 10 × 2 × 16) total m1.small instances. A bit high


Compute Considerations •  Bin Packing

–  Ensuring that compute resources are optimally used by right sizing resources –  Make sure to balance your vCPUs to Local Storage to Memory –  Example:

•  Each small instance is 1 vCPU, 2 GB Memory, 64 GB Ephemeral Storage •  Medium, large and X-Large are multiples of a small instance •  Server specification (Assuming dual socket 8 core with hyper-threading)

–  Total vCPUs = 2 x 8 x 2 = 32 vCPUs (Do take into account your over-commit ratio) –  Total memory = 32 x 2 GB Memory = 64 GB Memory –  Total ephemeral Storage = 32 x 64 GB storage = 2 TB Ephemeral storage


Compute Considerations • I/O Issues

– Don’t overload your compute nodes – 4 Socket servers probably not useful unless you have CPU intensive workloads and

will run small number of Instances per physical node – Consider 10GbE everywhere

• 10 Gbase-T is affordable for your management network and runs on Cat 6E • 10 GbE for everything else

• Financial Costs – Operations cost is key – automate as much as possible

• Easier management • Less error • Measure all metrics to resolve issues quick

– Cost $/instance, $/GB Storage – Commodity vs enterprise


Compute Considerations • Server Form Factor (1U vs 2U vs Sleds vs Blades)

– Affects rack density – DC Power requirements – Weight (Can your floors handle the weight) – Hardware support (NICs, HBA,# of disks etc)

• 1U limited in hardware support – The denser the rack the larger the networking requirements (Not for Blades) • Sleds will double or quadruple networking requirements and power requirements

• Denser – Consider potential bottle necks for blades – Expandability (Cards) – Automation complexities


Compute Considerations •  Security

–  Compute nodes have a number of cross domain access •  External, data, management and storage networks

–  Important that compute nodes are hardened appropriately •  Local Storage

–  Consider if you require local storage •  Do you want persistent storage or Ephemeral

–  Persistent for legacy or higher availability –  Ephemeral for cloud native or cost sensitive workloads

•  Networking –  Compute nodes require access to:

•  Management network •  Data Network •  Storage Network •  IPMI Network

–  Consider 10GbE TwinAx for Data and storage to ToR Switch –  Consider 10GBase-T for management to support increasing monitoring and log traffic –  Consider 100Mbps or 1GbE for IPMI



• Hypervisor support matrix – Choose a well supported hypervisor (KVM is best supported) – http://docs.openstack.org/developer/nova/support-matrix.html


Compute Considerations • Hypervisor Criteria

–  Group A •  These drivers are fully supported. Test coverage includes:

–  unit tests that gate commits –  functional testing that gate commits

• Drivers in this group include: –  libvirt (qemu/KVM on x86)

–  Group B •  These drivers are in a bit of a middle ground. Test coverage includes:

–  unit tests that gate commits –  functional testing providing by an external system that does not gate commits, but advises patch authors and reviewers of results

in gerrit (the code review system). • Drivers in this group include:

–  Hyper-V –  Vmware –  XenServer –  Xen via libvirt



•  Hypervisor Criteria –  Group C •  These drivers have minimal testing and may or may not work at any given time. Use them at your own risk. Test coverage includes: –  (maybe) unit tests that gate commits –  no public functional testing

•  Drivers in this group include. –  Baremetal –  Docker

–  LXC via libvirt


Rack Configurations •  This is a sample of how a full cloud would be

setup (Larger Scale). –  2 Management racks for high availability of

controller and management nodes and network aggregation. Each rack is a failure zone. 3 Racks if you want to use Pacemaker with Quorum

–  2 or more compute node racks –  2 or more storage node racks –  3 racks Ceph node racks, each rack is a failure

zone –  5 Racks Swift node racks, each rack is a failure

zone.

•  This setup can be partially filled to allow for future capacity.

•  Having this configuration follows best practices and prevents future migrations or reconfigurations. –  Allows for easy expansion planning

One Year Full Capacity

Network Aggregatio

n

Management

Cloud Manageme

nt

Network Aggregatio

n

Management

Cloud Manageme

nt

Compute

Network

Management

SAN iSCSI 3Par

NetAPP

Network

Management

Ceph Storage

Network

Management

Ceph Storage

Network

Management

Object Storage

Network

Management

Object Storage

Network

Management

Object Storage

Network

Management

Object Storage

Network

Management

Object Storage

Network

Management

Ceph Storage

Network

Management


Compute Considerations Scaling your cloud:


Network Considerations


Network Considerations • OpenStack Networking is complicated and has always had high-availability

issues – High-Availability issues are now at an acceptable level

• DVR • VRRP

• OpenStack Networking – Nova-Network (legacy)

• Flat – Each instance has an external IP address in common pool

• Vlan – Each tenant has a VLAN

•  4096 tenants

– Neutron •  first class networking service that gives full control over creation of virtual network resources to tenants • Via Tunneling or VLAN


Network Considerations • Topology

– Data + External Networks • These are your virtual networks • L2 Flat

– Legacy • L2 VLAN

– Not Scalable – Each Tenant has one or more VLANs

• Encapsulating – VXLAN – GRE – Scalable – SDN – Trend is moving in this direction

• Leaf and Spine architecture • Consider using 10GbE


Network Considerations •  Topology

– Storage Networks •  L2 network mostly •  Leaf and Spine architecture •  Consider using 10GbE

– Management Network •  L2 network mostly •  Leaf and Spine architecture •  Consider 10GBaseT if you have a large scale cloud

– IPMI/iLO •  IPMI is usually very insecure •  Advisable to not have it routable •  Access via management servers •  100Mbps (IPMI) or 1GbE


Network Considerations Leaf and Spine Architecture

Rack Rack Rack Rack

Leaf

Spine

Redundant and 2 Hops to any node


Network Considerations •  Layer 2

–  Pros •  Speed •  Reduced overhead •  No need for controller to keep track of where everything is

–  Cons •  VLAN = 4096 OpenStack tenants •  Number of MAC address storable on a switch •  MLAG is proprietary •  Difficult to troubleshoot •  ARP complicated in large networks •  Constant churn on MAC Tables and broadcast


Network Considerations •  Layer 3

•  L3 networks for the data network is the preferred trend via encapsulation technologies –  Pros

•  Resiliency and scalability •  Routing state change only occurs when there is failures •  Easy to troubleshoot •  QoS

–  Cons •  Must rely on L2 over L3 tunneling, which can effect Performance •  Requires L3 capable switch


Network Considerations • Encapsulating Technology • OpenStack Native Drivers

– Compute nodes use Open vSwitch (OVS) to create virtual networks – OVS maintains information about tenant networks – OVS wraps L2 packets into a L3 packet and send it out to the NIC – Compute nodes are configured with an L3 IP address and packets are sent via normal L3

mechnisms – Packets are decapsulated at the destination compute node.

• OpenStack Drivers –  GRE (Native) –  VXLAN (Native) –  MidoNet (3rd Party) –  OpenContrail (3rd Party) –  Nuage (3rd Party) –  NSX (3rd Party)


Network Considerations

• Security – Segregate different domains – Network misconfiguration

• Automate switch configurations – Single Point of Failure

• Having redundant ToR Switches is expensive • MBTF of a properly maintained switch is well past 5 years

– Consider having spares for failed switches

– Firewalls between network segments


Network Considerations •  Switch Configurations

–  Port Speed •  10GBase-T network switches prices are relatively cheap. Consider this for the management network in large scale clouds •  10GbE network switches for Data and storage networks •  40GbE uplinks

–  Cabling •  SFP+ TwinAx

–  Server to ToR switch (Unless latency is an issue) –  Cost is an issue

•  SFP+ Fiber – Between ToR and aggregation layer

–  Layer 3 routing for data network switches via ECMP


Network Considerations •  Switch Configurations

–  Switch Count •  This is dependent on Rack/Server density •  Try to keep it to one switch per network

–  Port Count/Density •  Depends on rack/server density •  Generally 48 or more ports for compute, management, block storage networks •  Object and block storage may require less port density if using large JBODs

–  Software •  New Linux based switches allow for easy integration with Automation tools •  Arista and Cumulus Linux

–  Cost •  Commodity vs Proprietary •  Port Type and speed •  Redundancy


Network •  Use Case Network Considerations

–  High availability •  Proper sizing of the network to maintain replication of data between sites for high availability or Disaster Recovery.

–  Big data •  Demand on network resources via data ingress and replication.

–  Virtual desktop infrastructure (VDI) •  Sensitive to network congestion, latency, jitter, and other network characteristics.

–  Voice over IP (VoIP) •  Sensitive to network congestion, latency, jitter, and other network characteristics. •  Symmetrical traffic pattern and requires QoS

–  Video Conference or web conference •  sensitive to network congestion, latency, jitter, and other network characteristics.

–  High performance computing (HPC) •  High east-west traffic patterns for distributed computing


Object Storage Considerations


Swift Object Storage Consideration

Storage Node

Storage Node

Storage Node

Storage Node

Storage Node

Storage Node

Storage Node

Storage Node

Storage Node

Storage Node

Storage Node

Storage Node

Storage Node

Storage Node

Storage Node

Zone 1 Zone 2 Zone 3 Zone 4 Zone 5

Proxy Node

Proxy Node

Proxy Node

Scale Out

Scale Out


Object Storage Considerations •  Goal

–  Maximize storage –  Keep cost per terabyte down

•  Large capacity Nearline or SATA disk only

•  No RAID

•  Swift handles integrity checks and replication

•  Maximize disk density –  60 disk JBODs in 4U –  12 3.5” disks in 1U –  30 disk OCP JBOD


Object Storage Considerations •  Minimum 5 zones (Generally at the rack level)

–  5 nodes to achieve quorum for durability and availability –  Zones are fault zones (Power, cooling, network, etc) –  3 replicas of every object blob

•  Consider separate replication and front-end networks

•  Use dual socket 8 core or more CPUs for storage nodes

•  Proxy nodes should also have a lot of memory and/or SSD drives for caching

•  Aim for 1PB per Rack

•  10GbE network for replication, front-end and public networks.

•  Distribute over availability zones or regions


Block Storage Considerations


Block Storage Considerations •  Understand the workloads and requirements that will drive the use of block storage in the cloud.

•  Create block storage tiers for most common workloads –  Normal Tier

•  Ceph •  SATA based SAN

–  High-Performance tier •  SSD based SAN •  Enhance Ceph with SSD caching

•  Local Storage –  Ephemeral (non-persistent, low cost, speed) –  Persistent (Data integrity, increase speed) –  File based

•  Img, VDH, VMDK, VDI, QCOW2, AMI

–  Block Based •  LVM


Local File based Ephemeral Storage

Host

Logical Volume

HDD 01 HDD 02 HDD 03 HDD 04 HDD 05 HDD 06 HDD 07 HDD 08 HDD 09 HDD 10 HDD 11 HDD 12

VM

APP

OS

Host File System VM Image VM Image VM Image

VM

APP

OS

VM

APP

OS

-  Images stored on the host file system as an image file -  1U or 2U with 4 – 24 disks

RAID 0/JBOD


Local File based Persistant Storage

Host

Logical Volume


VM

APP

OS

Host File System VM Image VM Image VM Image

VM

APP

OS

VM

APP

OS

-  Images stored on the host file system as an image file -  1U or 2U with 4 – 24 disks

RAID 5/RAID 6/RAID 10


Local Block based Ephemeral Storage

Host

Logical Volume


VM

APP

OS

Storage Block

VM

APP

OS

VM

APP

OS

-  VM backing store is generally allocated as logical Volumes -  1U or 2U with 4 – 24 disks

RAID 0/JBOD

Storage Block

Storage Block


Local Block based Persistent Storage

Host

Logical Volume


VM

APP

OS

Storage Block

VM

APP

OS

VM

APP

OS

-  VM backing store is generally allocated as logical Volumes -  1U or 2U with 4 – 24 disks


Storage Block

Storage Block


Block Storage Considerations •  Networked Storage

–  File Based •  NFS/NAS

–  NetAPP

•  Distributed FileSystem –  GlusterFS

–  Block Based •  SAN

–  iSCSI –  3Par –  StoreVirtual

•  Distributed Block Storage –  Ceph via RBD


Highly Availability NAS


Logical Volume

Host File System

Compute Host

-  Images stored on remote file system as an image file -  1U or 2U with JBODs

JBOD SIM/FC

VM

Imag

e

VM

Imag

e

VM

Imag

e

Storage Host

Logical Volume

Host File System


VM

Imag

e

VM

Imag

e

VM

Imag

e

APP

OS VM

APP

OS VM

APP

OS VM

HA/Fail-Over Storage Host

Network File System (NFS/CIFS) Heartbeat


Distributed Network Filesystem

Tend to be CPU intensive

JBOD JBOD

JBOD Mode

Logical Volume

File System

JBOD Mode

Logical Volume

File System

Storage Host Storage Host

JBOD JBOD

JBOD Mode

Logical Volume

File System

JBOD Mode

Logical Volume

File System


Distributed File System CephFS GlusterFS

Compute Host

VM

Imag

e

APP

OS VM

VM

Imag

e

APP

OS VM

VM

Imag

e

APP

OS VM

Compute Host

VM

Imag

e

APP

OS VM

VM

Imag

e

APP

OS VM

VM

Imag

e

APP

OS VM

Storage Pool

Image file is striped and replicated by factor of X (Sync/Async)

Cluster Management Server


Highly Available SAN


Compute Host

JBOD SIM/FC

Storage Host


APP

OS VM

APP

OS VM

APP

OS VM

Heartbeat

Logical Volume iSCSI FCoE AoE FC

Storage LUN

Storage LUN

Storage LUN

Logical Volume iSCSI FCoE AoE FC

Storage LUN

Storage LUN

Storage LUN

HA/Fail-Over Storage Host


Highly Available SAN - Replicated


Compute Host

-  Images stored on remote file system as an image file -  1U or 2U with JBODs

JBOD

Storage Host


APP

OS VM

APP

OS VM

APP

OS VM

Logical Volume iSCSI FCoE AoE Storage

LUN Storage

LUN Storage

LUN

Logical Volume iSCSI FCoE AoE Storage

LUN Storage

LUN Storage

LUN

JBOD

Heartbeat/Replication HA/Fail-Over Storage Host


Distributed Networked Block Storage

Storage Pool

SIM

JBOD

SIM

JBOD

JBOD Mode

Logical Volume

File System

JBOD Mode

Logical Volume

File System


SIM

JBOD

SIM

JBOD

JBOD Mode

Logical Volume

File System

JBOD Mode

Logical Volume

File System


Compute Host APP

OS VM

APP

OS VM

APP

OS VM

Compute Host APP

OS VM

APP

OS VM

APP

OS VM

Cluster Management Server

Storage LUN

Storage LUN

Storage LUN

Storage LUN

Storage LUN

Storage LUN

Replicated Copy on Write (COW) Gold VM

Replicated COW VM Data (Delta of Gold VM)

Replicated VM Block

Striped and Replicated VM Block


Block Storage Considerations •  Well Supported block storage systems

–  Linux iSCSI –  Ceph –  NetAPP –  3Par –  StoreVirtual


Block Storage Considerations •  Consider distributed block storage for normal workloads

–  Easy to add capacity –  May benefit from distributed reads

•  Connectivity –  10GbE vs FC (review cost and salability requirements)

•  Instance and image locations –  Some block storage can store base images permanently for fast loading (CoW)

•  E.g Ceph

•  Server hardware –  Commodity/CoTs vs Proprietary

•  Cost •  Features (High-Availability, snapshots, etc)


Block Storage Considerations •  Capacity

–  Are there capacity limits

•  RAID –  Redundancy –  Performance –  Watch out for rebuild times for large cluster –  N+2 is recommended

•  Benchmark potential storage systems

•  Maintenance Tasks –  Disk replacements without interuption –  Scrubbing data


Operational Considerations


Software Considerations •  Choice of operating system

–  Linux •  RHEL •  Ubuntu •  SuSe •  CentOS •  Debian •  Windows

•  Choice of hypervisor –  KVM, Xen, VMWare ESX, HyperV, LXC, baremetal, docker

•  Monitoring & Metrics –  Measure everything –  Nagios still heavily used

•  Xabbix, Sensu, reimann,

•  Log Management –  Elastic Search, logstash, kibana

•  Automation –  Chef, Puppet, CSA, CSA-Chef, server automation


Software Considerations

Advice

Automate, Automate, Automate and Automate


Software Considerations •  Capacity Planning

–  Measuring metrics is important to understand utilization trends and will help plan your future capacity plans

•  Selection of supplemental software –  Message Queue, Databases –  Load balancers –  HA –  Memcache/Redis –  Monitoring –  Log management –  Metrics

•  Keep things as standardized as possible to simplify management


Software Considerations •  Cost

•  Supportability

•  Management tools

•  Scale and performance

•  Security

•  Supported features

•  Interoperability


OpenStack Considerations •  Compute (Nova) •  Image Service (Glance)

•  Object Storage (Swift)

•  Dashboard (Horizon)

•  Identity Service (Keystone) •  Networking (Neutron)

•  Block Storage (Cinder)

•  Orchestration (Heat)

•  Telemetry (Ceilometer) •  Database (Trove)

•  Elastic Map Reduce (Sahara)

•  Bare Metal Provisioning (Ironic)

•  Multiple Tenant Cloud Messaging (Zaqar) •  Shared File System Service (Manila)

•  DNSaaS (Designate)

•  Security API (Barbican)


Availability Considerations •  Most HA issues have been solved to acceptable levels

–  Biggest issues have been networking •  DVR + VRRP

–  Ceilometer •  Greatly improving

•  Switches, routes and redundancies of power should be factored into core infrastructure, as well as the associated bonding of networks to provide diverse routes to your highly available switch infrastructure.

•  Backups – Git and databases

•  Virtual Infrastructure Availability –  Mult-region/AZ deployment –  Scale out design


Security Considerations •  Cross Domain access

•  Normal security applies

•  Identity and access control

•  SSL termination

•  Encrypted communications


Thank you

Contact information

OpenStack Architectural Decisions · – Total vCPUs = 2 x 8 x 2 = 32 vCPUs (Do take into account...

Documents

Transcript of OpenStack Architectural Decisions · – Total vCPUs = 2 x 8 x 2 = 32 vCPUs (Do take into account...