OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf ·...

26
OpenStack Service High Availability Openstack Pakistan meetup, April 2016 Syed Affan Ahmed Director Engineering, PLUMgrid

Transcript of OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf ·...

Page 1: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

OpenStack Service High AvailabilityOpenstack Pakistan meetup, April 2016

Syed Affan Ahmed

Director Engineering, PLUMgrid

Page 2: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

• OpenStack and its Services

• High Availability in Openstack: Overview

• Services HA

• Keystone

• Glance

• Cinder

• Nova

• HA deployment architecture (one type)

Agenda

Page 3: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

High Availability in OpenStack

Page 4: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

• Designed to minimize two aspects

• System downtime

• Data loss

Both in the context of single-point-of-failure and cascading set of events

High Availability Principles

Page 5: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

• OpenStack Services HA (our focus now)• Architectural support

• Important if a public cloud offering is part of ROI

• Application HA• Cloud native (scalable)

• Legacy (pets) • Similar to Services HA

• Load-balanced

• Shared storage and hypervisor-based

Types of HA

Page 6: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

• Stateless

• Provides a response to each individual request

• nova-api, nova-conductor, glance-api, keystone-api, neutron-api and nova-

scheduler

• HA achieved via redundant instances & load-balancer

• Stateful

• Provides a response to a request that is based on previous requests

• OpenStack databases & message queue

• HA requires more substantial configuration

Stateless vs Stateful Servicesand the impact on complexity

Page 7: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

• Active/Passive brings additional resources online to replace those

that have failed

• Single master

• Active/Active has all resource being used concurrently

• Multi-master

A/P and A/A HA ConfigurationDesign choices

Page 8: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Openstack Services HA An illustrative example

Page 9: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

• Pacemaker • Cluster management

• Virtual IP (can also be done with keepalived)

• STONITH

• HAProxy• Load balancing and service failure detection

Two key components

Page 10: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

Give Me Horizon

Web UI Now!

Horizon Controller

HA illustratedFullsuit edition!

Inspired by Arthur Berezin’s talk : “Deep Dive into Highly Available OpenStack Architecture”

Page 11: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

Give Me Horizon

Web UI Now!

Horizon Controller

Single Point of Failure

Page 12: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

Give Me Horizon

Web UI Now!

HAProxy Controller 1

Horizon Controller 1 Horizon Controller 1 Horizon Controller 1

Page 13: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

Give Me Horizon

Web UI Now!

HAProxy Controller 1

Horizon Controller 1 Horizon Controller 2 Horizon Controller 3

Single Point of Failure

Each Could Fail

Page 14: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

Give Me Horizon

Web UI Now!

HAProxy Controller 1

Horizon Controller 1 Horizon Controller 2 Horizon Controller 3

Single Point of Failure

Pacemaker Cloned Horizon Service

Page 15: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

Give Me Horizon

Web UI Now!

Horizon Controller 1 Horizon Controller 2 Horizon Controller 3

Pacemaker Cloned Horizon Service

HAProxy Controller 1 HAProxy Controller 2 HAProxy Controller 3

Pacemaker Cloned HAProxy Service

Page 16: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

Give Me Horizon

Web UI Now!

Horizon Controller 1 Horizon Controller 2 Horizon Controller 3

Pacemaker Cloned Horizon Service

HAProxy Controller 1 HAProxy Controller 2 HAProxy Controller 3

Pacemaker Cloned HAProxy ServiceHorizon

VIP

Page 17: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Services HA

All diagrams inspired from Arthur’s talk!

Page 18: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

• Cloned Stateless HTTPD Service

• Same SSL Certs on all nodes

• Cache is local on each host

Keystone HA

ClonedHAProxy HAProxy

ClonedHTTPd/

Keystone

HTTPd/

Keystone

Keystone

VIP

pcsd

Node 1

pcsd

Node 2

STONITH STONITH

Page 19: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

•Both services (API and Registry) are Cloned Active/Active • Both are load balanced

and VIP-ed

• Active VIP on different nodes

Glance HA

ClonedHAProxy HAProxy

ClonedGlance

Registry

Glance

Registry

Glance

API

VIP

pcsd

Node 1

pcsd

Node 2

STONITH STONITH

Glance

Registry

VIP

ClonedGlance-API Glance-API

Page 20: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

•Cinder-API, scheduler are Stateless•Cloned, LB and VIP

•Cinder-Volume is A/P due to potential race conditions

Cinder HA

Storage

ClonedHAProxy HAProxy

Cinder

API

VIP

pcsd

Node

1

pcsd

Node

2STONITH STONITH

ClonedCinder-API Cinder-API

ClonedScheduler Scheduler

A/PVolume VolumeDriver Driver

Page 21: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

• Nova-API configured with LB and VIP

• Nova-API, Nova-Scheduler and Nova-Conductor are Stateless A/A cloned services

Nova Controller HA

ClonedHAProxy HAProxy

ClonedConductor Conductor

Nova

API

VIP

pcsd

Node

1

pcsd

Node

2STONITH STONITH

ClonedNova-API Nova-API

ClonedScheduler Scheduler

Page 22: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

• Provider specific, and typically a SDN solution

• PLUMgrid most scalable solution out there• Fully replicated Control Plane and

• a fully distributed Data Plane

Neutron HA

Page 23: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

A full Services HA deploymentRed Hat reference Architecture

Page 24: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

Copyright © PLUMgrid, Inc. 2011-2016

A full DCN deployment the above HA Architecture

Page 25: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both
Page 26: OpenStack Service High Availabilityopenstackpakistan.github.io/web/resources/meetup_2nd_HA.pdf · OpenStack Service High Availability Openstack Pakistan meetup, April 2016 ... Both

(408) 800-7586 www.plumgrid.com

5155 Old Ironsides Dr.

Suite 200

Santa Clara, CA 95054,THANK YOU!Keep in Touch and Contact Us