1

Run Kubernetes on OpenStack and Bare Metal

Fast

Ramon Acedo RodriguezSenior Principal Product Manager, Red Hat

OPEN

INFRASTRUCTURE SUM

MIT | SHAN

GHAI, NOVEM

BER 4-6 2019

2

OPTIO

NAL SECTION

MARKER O

R TITLE

Open Hybrid CloudVision

3 BARE METAL VIRTUAL PRIVATE CLOUD

The Open Hybrid CloudThe 4 Footprints

PHYSICAL VIRTUALPRIVATECLOUD

PUBLICCLOUD

APP APP APP APP

Goal: Give developers the freedom to innovate faster across on-premises and public clouds

4 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

PHYSICAL

APP APP APP APP

VIRTUALPUBLICCLOUD

VM CC C

On-premises

Kubernetes-powered Open Hybrid Cloud

5 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

VIRTUAL

Containers | VMs | Serverless Apps

BARE METAL PRIVATE CLOUD PUBLIC CLOUDS

Kubernetes on OpenStack is focused on the private cloud

6

Kubernetes on OpenStack

Introduction

7 VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

DATACENTRE

WORKLOADDRIVEN

PROGRAMMATIC API DRIVEN

ACROSS INFRASTRUCTURE

DEEPLYINTEGRATED

Why Kubernetes on OpenStack?Open, scalable, managed, workloads.

SCALE OUT

SOLID FOUNDATION

8

Kubernetes on OpenStack Integrations

Kubernetes and OpenStack Are Complementary

Consumption of resourcesProvides the container platform layer

Exposition of resourcesProvides the infrastructure layer

complementary

Kubernetes on OpenStack Key Integration Points

Kubernetes on OpenStack Integration Example: Cinder

OpenShift on OpenStack Logical Architecture

13

Installation RequirementsOpenShift on OpenStack

Requirements in your OpenStack Platform

Red Hat OpenStack Platform 13 (Queens)

3 Master nodes● At least 16 GB RAM, 4 vCPUs and 25 GB Disk

At least 2 worker nodes● A least 8 GB RAM, 2 vCPUs and 25 GB Disk

Object Storage (Swift)

CoreOS image

OpenStack Resources:● Floating IPs: 2● Security Groups: 3● Security Group Rules: 60● Routers: 1● Subnets: 1● RAM: 112 GB● vCPUs: 28● Volume Storage: 175 GB● Instances: 7● Swift containers: 2● Swift objects: 1● Available space in Swift: at least 10 MB

https://github.com/openshift/installer/blob/master/docs/user/openstack/README.md

15

OpenShift Internal Load Balancing and DNS

DNS

● CoreDNS and mDNS run on all nodes

Internal DNS and Load Balancing

Load Balancing

● HAProxy + keepalived on master nodes

● Use of keepalived VIPs for:

○ Internal API○ Ingress traffic to workloads○ Internal DNS requests

The openshift-installer will configure internal Load Balancing & DNS

https://github.com/openshift/installer/blob/master/docs/design/openstack/networking-infrastructure.md

17

Kuryr-Kubernetes SDNSDN for OpenShift on OpenStack

Kuryr improves the network performance of pods when running on OpenStack.

SDN solution using Kubernetes Container Network Interface (CNI) and OpenStack Neutron.

Provides interconnectivity between Kubernetes pods and OpenStack virtual instances

Kuryr-Kubernetes Logo

Kuryr-Kubernetes CNI

18

Recommended when

● Neutron tenant networks are used for OpenShift Application Nodes

Not recommended when

● Using only OpenStack VLAN-based provider networks (Kuryr not needed)

● Many OpenShift services and few hypervisors:○ For each OpenShift service, Octavia will run a VM with a

load balancer in it

When to use Kuryr

19

Requires

● OpenStack Queens or newer● Octavia Load Balancer● Neutron Trunk Ports

Kuryr Internal Architecture

20

Kuryr integrates with Neutron and Octavia

Pods are connected via Kuryr CNI to Neutron

Kuryr creates Load Balancers with Octavia for the Kubernetes services

21

Bandwidth tests between pods on the same hypervisor show slightly better performance

Kuryr Performance ComparisonPerformance test done with OpenShift 3.11 / Kubernetes 1.11 and OpenStack Queens

22

Bandwidth tests between pods on different hypervisors show up to 9x performance improvement

Kuryr Performance ComparisonPerformance test done with OpenShift 3.11 / Kubernetes 1.11 and OpenStack Queens

23

Kuryr Performance Comparison Blog PostPerformance test done with OpenShift 3.11 and OpenStack Queens

https://blog.openshift.com/accelerate-your-openshift-network-performance-on-openstack-with-kuryr

24

Reference Architecture

OpenShift on OpenStack Reference ArchitectureCurrent: OSP 13 LTS and OCP 3.11 with Kuryr | Next: OSP 13 LTS and OCP 4.3 with Kuryr

25 https://access.redhat.com/documentation/en-us/reference_architectures/2019/html-single/deploying_red_hat_openshift_container_platform_3.11_on_red_hat_openstack_platform_13

26

Ways to Install Kubernetes on

OpenStackIntroduction

27 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

VIRTUALBARE METAL PRIVATE CLOUD

Full Stack AutomationSimplified, opinionated

Installer-provisioned OpenStack resources (IPI)

Pre-existing InfrastructureUser-provisioned

OpenStack resources (UPI)

Kubernetes Installation with OpenShift 4 on OpenStackInstallation Experiences

28 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

VIRTUALBARE METAL PRIVATE CLOUD PUBLIC CLOUDS

Installer Provisions:

Networks

Internal Load Balancers1

Internal DNS1

OpenStack Instances

Red Hat CoreOS

CoreOS Ignition Configs

OpenShift Nodes

OpenShift Cluster Resources

Full Stack Automation InstallationDeploying OpenShift on OpenStack with installer-provisioned infrastructure (IPI)

Red Hat OpenStack Platform

Overcloud

OpenShift Cluster

. . .

openshift-install

1External Load Balancers (routing) and external DNS servers are provided by the user

29 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

VIRTUALBARE METAL

User Provisions:

Networks

Internal Load Balancers1

Internal DNS1

OpenStack Instances

Red Hat CoreOS

CoreOS Ignition Configs

OpenShift Nodes

OpenShift Cluster Resources

Installing on Pre-existing InfrastructureDeploying OpenShift on OpenStack with user-provisioned infrastructure (UPI)

openshift-install

User provisioned resources

Red Hat OpenStack Platform

Overcloud

OpenShift Cluster

. . .

30 VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

Installing on OpenStack DocumentationUser-provisioned Infrastructure | Installer-provisioned Infrastructure

USER PROVISIONED INFRASTRUCTURE

INSTRALLER- PROVISIONED INFRASTRUCTURE

https://github.com/openshift/installer/blob/master/docs/user/openstack/README.mdhttps://github.com/openshift/installer/blob/master/docs/user/openstack/install_upi.md

OpenShift 4.2 on OpenStack

31 try.openshift.com

32

OpenStack Bare MetalIronic Introduction

Bare Metal On-Trend

OpenStack User Survey 2017

Among users who run Kubernetes on OpenStack, adoption of Ironic is even stronger with 37% relying on it.

OpenStack User Survey 2018

Popular Use Cases

Kubernetes on Bare Metal

High-Performance Computing

Direct Access to Dedicated Hardware Devices

Big Data and Scientific Applications

Bare Metal On-Trend

OpenStack Bare Metal Service - IronicHardware Lifecycle Management

Hardware InspectionServers and Network Switches (via LLDP)

OS Image ProvisioningSupporting qcow2 images

Routed Spine/Leaf NetworkingProvisioning over routed networks

Multi-TenancyWith network isolation between tenants

Node Auto-discovery

Broad Power Management SupportRedfish, iDrac, iRMC, iLo, IPMI, oVirt, vBMC

36

OpenStack Bare MetalFeatures

Tenant-Dedicated NetworksInstead of a shared flat network

Provisioning Over an Isolated Network

Switch Ports Dynamically ConfiguredAt deployment time and on termination

Link Aggregation L2 Switch

BM

NIC NIC

LAG

bond

Configured by ML2 plug-in

Configured by cloud-init using

metadata

L2 Switch

BM

NIC

VLANs set by by ML2 plug-in

BM

NIC

L2 Switch

Multi-Tenant Support with Isolation Between TenantsAvailable from OpenStack Queens

Multi-Tenant Support with Isolation Between TenantsAvailable from OpenStack Queens

Neutron ML2 Networking-Ansible Driver

Multiple Switch Platforms in a Single ML2 DriverLeveraging the Networking Ansible modules

Available in OpenStack Queens | Red Hat OpenStack Platform 13OSP 13 Long Life Support

Provisioning Network is configured in the switch

Boot BM on Tenant

Network

ML2 Plug-in Configures

Switch

BM is Provisioned

ML2 Plug-in Configures

Switch

Tenant Network is configured in the switch

BM is ready

L2 Switch

BMNIC

BMNIC

spine switch

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

spine switch spine switch

L3 routed networks

ToR/leaf switch

Bare Metal

Ironic Node

Ironic Node

Ironic Node

Bare Metal

ToR/leaf switch ToR/leaf switch

DHCP Relay DHCP Relay DHCP Relay

L3 routed networks

L3 Spine and Leaf TopologiesIronic provisioning bare metal nodes over routed networks

DHCP RelayAllowing PXE booting over L3 routed networks

L3 Routed Networks (Spine/Leaf Network Topologies)Available from OpenStack Queens IPv6 Support being added to OpenStack Train and above

BIOS Configurationsdocs.openstack.org/ironic/latest/admin/bios.html

Get and Set BIOS SettingsRetrieve and apply BIOS settings via CLI or REST API. The desired BIOS settings are applied during manual cleaning.

Settings Applied During Node CleaningThe desired BIOS settings are applied during manual cleaning

[{ "name": "hyper_threading_enabled”, "value": "False" }, { "name": "cpu_vt_enabled", "value": "True" }]

Ironic Inspector Nodes Auto-Discovery

Just Power On the NodesNodes PXE boot from the provisioning network used by Ironic

Automatic Node InspectionNodes boot from the network and their hardware is inspected

Nodes Automatically Registered with IronicAfter inspection they are registered with Ironic and ready to be deployed

Use Rules to Set Node Properties E.g. set Ironic driver (iDrac, Redfish…) based on inspection data, set BMC credentials, etc.

cat > rules.json << EOF[ { "description": "Set the vendor driver for Dell hardware", "conditions": [ {"op": "eq", "field": "data://auto_discovered", "value": true}, {"op": "eq", "field": "data://inventory.system_vendor.manufacturer", "value": "Dell Inc."} ], "actions": [ {"action": "set-attribute", "path": "driver", "value": "idrac"}, {"action": "set-attribute", "path": "driver_info/drac_username", "value": "root"}, {"action": "set-attribute", "path": "driver_info/drac_password", "value": "calvin"}, {"action": "set-attribute", "path": "driver_info/drac_address", "value": "{data[inventory][bmc_address]}"} ] }]EOF

$ openstack baremetal introspection rule import rules.json

Data collected during inspection

E.g: Use the the idrac driver and its credentials if a Dell node is detected

Redfish Support in Ironic

API-driven Remote Management PlatformManage large amounts of physical nodes via API. redfish.dmtf.org

Included in Modern BMCsMost vendors support Redfish in the latest models

Supported in IronicIntroduced in OpenStack Pike, along with the Sushy library

Improvements in OpenStack Train and BeyondOut-of-band inspection of nodes, boot from virtual media (without DHCP) and BIOS configurations

openstack baremetal node create \ --driver redfish \ --driver-info redfish_address=https://example.com \ --driver-info redfish_system_id=/redfish/v1/Systems/CX34R87 \ --driver-info redfish_username=admin \ --driver-info redfish_password=password

43

OpenShift on OpenStack-Managed

Bare MetalDeployment of Kubernetes on the metal

Kubernetes Cluster

Kubernetes on Bare MetalDeploy Kubernetes on OpenStack-managed bare metal nodes

OpenShift Installer

Master Node

Worker Node

Worker Node

Deploy OpenShift

OpenStack with Ironic

https://docs.openshift.com/container-platform/4.2/installing/installing_bare_metal/installing-bare-metal.html

45

Metal3Deployment of Kubernetes on the metal

46 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

VIRTUAL

Containers | Serverless Apps

BARE METAL PRIVATE CLOUD PUBLIC CLOUDS

Metal3 is focused on Kubernetes on bare metal

Metal3

Metal Kubed, bare metal host provisioning for Kubernetes

Metal3 Kubernetes Operators FrameworkIronic

Metal3 enables bare metal host management with Kubernetes.

48

Metal3 runs on Kubernetes.

And is managed through Kubernetes interfaces.

Metal3

Metal Kubed, bare metal host provisioning for Kubernetes

49

Actuator allows Kubernetes to get Machines the same way it would in a public cloud, using the Kubernetes cluster-api

Operator uses Ironic behind the scenes to manage the physical hardware represented as BareMetalHost objects.

MACHINE CONTROLLER

BARE-METAL ACTUATOR

BARE METAL OPERATOR

Bare Metal Management PodsBare Metal Management PodsBARE-METAL MANAGEMENT PODS

Metal3

Metal Kubed, bare metal host provisioning for Kubernetes

Ironic

50

Try it:

http://metal3.io/try-it.html

Install OpenShift using it:

https://github.com/openshift/installer/blob/master/docs/user/metal/install_ipi.md

Metal3

Metal Kubed, bare metal host provisioning for Kubernetes

51

SummaryKubernetes on OpenStack

52 VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

Run Kubernetes on OpenStack and Bare Metal Fast

Install Kubernetes on OpenStackhttps://github.com/openshift/installer/blob/master/docs/user/openstack/README.mdhttps://github.com/openshift/installer/blob/master/docs/user/openstack/install_upi.md

Kuryr performancehttps://blog.openshift.com/accelerate-your-openshift-network-performance-on-openstack-with-kuryr

Reference Architecturehttps://access.redhat.com/documentation/en-us/reference_architectures/2019/html-single/deploying_red_hat_openshift_container_platform_3.11_on_red_hat_openstack_platform_13

Ironic Bare Metalhttps://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/bare_metal_provisioning/index

Metal3 metal3.io

53

Red Hat is the world’s leading provider of enterprise

open source software solutions. Award-winning

support, training, and consulting services make Red

Hat a trusted adviser to the Fortune 500.

Thank you