Operations Guide CLM - SUSE Linux · 5.4 Identity Service Token Validation Example126 5.5...

Operations Guide CLM

Operations Guide CLM

At the time of the SUSE OpenStack Cloud 9 release, this guide contains informationpertaining to the operation, administration, and user functions of SUSE OpenStackCloud. The audience is the admin-level operator of the cloud.

Publication Date: 07/27/2020

SUSE LLC1800 South Novell PlaceProvo, UT 84606USA

https://documentation.suse.com

Copyright © 2006– 2020 SUSE LLC and contributors. All rights reserved.

Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 Li-

cense : https://creativecommons.org/licenses/by/3.0/legalcode .

For SUSE trademarks, see https://www.suse.com/company/legal/ . All other third-party trademarks are the

property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its

affiliates. Asterisks (*) denote third-party trademarks.

All information found in this book has been compiled with utmost attention to detail. However, this does

not guarantee complete accuracy. Neither SUSE LLC, its affiliates, the authors nor the translators shall be

held liable for possible errors or the consequences thereof.

https://documentation.suse.comhttps://creativecommons.org/licenses/by/3.0/legalcodehttps://www.suse.com/company/legal/

Contents

1 Operations Overview 11.1 What is a cloud operator? 1

1.2 Tools provided to operate your cloud 1

1.3 Daily tasks 2

1.4 Weekly or monthly tasks 3

1.5 Semi-annual tasks 4

1.6 Troubleshooting 4

1.7 Common Questions 5

2 Tutorials 62.1 SUSE OpenStack Cloud Quickstart Guide 6

Introduction 6 • Overview of

components 6 • Preparation 7 • Getting Started 7

2.2 Installing the Command-Line Clients 12Installing the CLI tools using the input model 12 • Installing the CLI tools

using Ansible 14

2.3 Cloud Admin Actions with the Command Line 15Creating Additional Cloud Admins 15 • Command Line

Examples 15 • Assigning the default service admin

roles 19 • Customize policy.json on the Cloud Lifecycle

Manager 26 • Roles 26

2.4 Log Management and Integration 27Overview 27 • The ELK stack 27 • Using the Elasticsearch

API 28 • For More Information 29 • Forwarding your logs 29

2.5 Integrating Your Logs with Splunk 31Integrating with Splunk 31

iii Operations Guide CLM

2.6 Integrating SUSE OpenStack Cloud with an LDAP System 31Configure your LDAP source 32

3 Cloud Lifecycle Manager Admin UI User Guide 363.1 Accessing the Admin UI 36

3.2 Admin UI Pages 37Services 37 • Packages 38 • Configuration 39 • Model 42 • Roles 46 • Servers 47 • Admin

UI Server Details 48

3.3 Topology 50Control

Planes 50 • Regions 52 • Services 53 • Networks 55 • Servers 57 • Roles 58

3.4 Server Management 60Adding Servers 60 • Activating Servers 71 • Deactivating

Servers 72 • Deleting Servers 76

3.5 Server Replacement 78Control Plane Servers 78 • Compute Servers 81

4 Third-Party Integrations 924.1 Splunk Integration 92

What is Splunk? 92 • Configuring Splunk to receive log messages from SUSE

OpenStack Cloud 9 92 • Forwarding log messages from SUSE OpenStack

Cloud 9 Centralized Logging to Splunk 93 • Searching for log messages from

the Spunk dashboard 95

4.2 Operations Bridge Integration 95

4.3 Monitoring Third-Party Components With Monasca 96monasca Monitoring Integration Overview 96 • monasca

Agent 96 • Writing Custom Plugins 97 • Configuring Check

Plugins 104 • Metric Performance Considerations 107 • Configuring

Alarm Definitions 107 • Openstack Integration of Custom Plugins into

monasca-Agent (if applicable) 113

iv Operations Guide CLM

5 Managing Identity 1145.1 The Identity Service 114

Which version of the Identity service should you

use? 114 • Authentication 114 • Authorization 116

5.2 Supported Upstream Keystone Features 117OpenStack upstream features that are enabled by default in SUSE OpenStack

Cloud 9 117 • OpenStack upstream features that are disabled by default

in SUSE OpenStack Cloud 9 119 • Stack upstream features that have been

specifically disabled in SUSE OpenStack Cloud 9 120

5.3 Understanding Domains, Projects, Users, Groups, and Roles 122Domains, Projects, Users, Groups, and

Roles 122 • Domains 123 • Domain

Administrator 123 • Projects 125 • Users and

Groups 125 • Roles 125

5.4 Identity Service Token Validation Example 126

5.5 Configuring the Identity Service 128What is the Identity service? 128 • Which

version of the Identity service should you

use? 128 • Authentication 129 • Authorization 131 • Default

settings 132 • Preinstalled roles 133

5.6 Retrieving the Admin Password 133Retrieving the Admin Password 133

5.7 Changing Service Passwords 134Password Strength 135 • Telling the configuration

processor which password(s) you want to

change 135 • private_data_metadata_ccp.yml 135 • Steps to

change a password 137 • Specifying password value 139 • Running

the configuration processor to change passwords 140 • Password

change playbooks and tables 141 • Changing RADOS Gateway

Credential 144 • Immutable variables 145

v Operations Guide CLM

5.8 Reconfiguring the Identity service 145Updating the keystone Identity Service 145 • Updating the Main

Identity service Configuration File 146 • Enabling Identity Service

Features 147 • Fernet Tokens 148

5.9 Integrating LDAP with the Identity Service 149Integrating with an external LDAP server 149 • Set up domain-specific

driver configuration - file store 150 • Set up or switch to domain-specific

driver configuration using a database store 161 • Domain-specific driver

configuration. Switching from a database to a file store 164 • Update LDAP

CA certificates 166 • Limitations 167

5.10 keystone-to-keystone Federation 169What Is Keystone-to-Keystone Federation? 169 • Setting Up a keystone

Provider 172 • Test It Out 178 • Inside keystone-to-keystone

Federation 179 • Additional Testing Scenarios 180 • Known Issues and

Limitations 192 • Scope Federated User to Domain 193

5.11 Configuring Web Single Sign-On 194What is WebSSO? 194 • Limitations 194 • Enabling

WebSSO 195 • Prerequisites 197 • WebSSO Using OpenID Method 208

5.12 Identity Service Notes and Limitations 213Notes 213 • Limitations 214 • keystone-to-keystone

federation 216 • System cron jobs need setup 218

6 Managing Compute 2196.1 Managing Compute Hosts using Aggregates and Scheduler Filters 219

Creating a nova Aggregate 219 • Using nova Scheduler Filters 221

6.2 Using Flavor Metadata to Specify CPU Model 224Editing the flavor metadata in the horizon dashboard 225

6.3 Forcing CPU and RAM Overcommit Settings 228Changing the overcommit ratios for your entire environment 230

6.4 Enabling the Nova Resize and Migrate Features 231Enabling Nova Resize and Migrate 231 • Disabling Nova Resize and

Migrate 232

vi Operations Guide CLM

6.5 Enabling ESX Compute Instance(s) Resize Feature 232Procedure 233

6.6 GPU passthrough 234Preparing nova via the input model updates 235 • Create a flavor 239

6.7 Configuring the Image Service 239How to enable glance image caching 240 • Allowing the glance copy-from

option in your environment 241

7 Managing ESX 2437.1 Networking for ESXi Hypervisor (OVSvApp) 243

More Information 246

7.2 Validating the neutron Installation 246

7.3 Removing a Cluster from the Compute Resource Pool 247Prerequisites 247 • Removing an existing cluster from the compute resource

pool 248 • Cleanup monasca-agent for OVSvAPP Service 250 • Removing

the Compute Proxy from Monitoring 253 • Cleaning the monasca Alarms

Related to ESX Proxy and vCenter Cluster 255

7.4 Removing an ESXi Host from a Cluster 257Prerequisite 257 • Procedure 258 • Clean up neutron-agent for OVSvAPP

Service 261 • Clean up monasca-agent for OVSvAPP Service 262 • Clean

up the entries of OVSvAPP VM from /etc/host 266 • Remove the OVSVAPP

VM from the servers.yml and pass_through.yml files and run the Configuration

Processor 266 • Clean Up nova Agent for ESX Proxy 267 • Clean Up

monasca Agent for ESX Proxy 267 • Clean Up ESX Proxy Entries in /etc/

host 268 • Remove ESX Proxy from servers.yml and pass_through.yml

files; run the Configuration Processor 268 • Remove Distributed Resource

Scheduler (DRS) Rules 269

7.5 Configuring Debug Logging 272To Modify the OVSVAPP VM Log Level 272 • To Enable OVSVAPP Service for

Centralized Logging 273

7.6 Making Scale Configuration Changes 274

7.7 Monitoring vCenter Clusters 275

vii Operations Guide CLM

7.8 Monitoring Integration with OVSvApp Appliance 278Processes Monitored with monasca-agent 278 • How It Works 278

8 Managing Block Storage 2798.1 Managing Block Storage using Cinder 279

Setting Up Multiple Block Storage Back-ends 279 • Creating a Volume Type

for your Volumes 280 • Managing cinder Volume and Backup Services 282

9 Managing Object Storage 2859.1 Running the swift Dispersion Report 285

Configuring the swift dispersion populate 285 • Running the swift dispersion

report 287

9.2 Gathering Swift Data 287Notes 288 • Using the swift-recon Command 288

9.3 Gathering Swift Monitoring Metrics 289Optional Parameters 291

9.4 Using the swift Command-line Client (CLI) 292

9.5 Managing swift Rings 294Rebalancing Swift Rings 295 • Using the Weight-Step Attributes

to Prepare for Ring Changes 296 • Managing Rings Using swift

Playbooks 298 • Determining When to Rebalance and Deploy a New

Ring 306 • Applying Input Model Changes to Existing Rings 308 • Adding

a New Swift Storage Policy 315 • Changing min-part-hours in

Swift 318 • Changing Swift Zone Layout 319

9.6 Configuring your swift System to Allow Container Sync 322Notes and limitations 322 • Prerequisites 324 • Configuring container

sync 324 • Configuring Intra Cluster Container Sync 327

10 Managing Networking 33010.1 SUSE OpenStack Cloud Firewall 330

Overview of the SUSE OpenStack Cloud Firewall configuration 331 • SUSE

OpenStack Cloud 9 FWaaS Configuration 331 • Making Changes to the

Firewall Rules 335 • More Information 336

viii Operations Guide CLM

10.2 Using VPN as a Service (VPNaaS) 337Prerequisites 337 • Considerations 337 • Configuration 338 • More

Information 342

10.3 DNS Service Overview 343For More Information 343 • designate Initial Configuration 343 • DNS

Service Monitoring Support 346

10.4 Networking Service Overview 348Installing the Networking Service 348 • Working with the Networking

service 348 • Reconfiguring the Networking service 348 • For

more information 349 • Neutron External Networks 349 • Neutron

Provider Networks 352 • Using IPAM Drivers in the Networking

Service 359 • Configuring Load Balancing as a Service (LBaaS) 369 • Load

Balancer: Octavia Driver Administration 371 • Role-based Access

Control in neutron 383 • Configuring Maximum Transmission Units in

neutron 395 • Improve Network Peformance with Isolated Metadata

Settings 401 • Moving from DVR deployments to non_DVR 402 • OVS-

DPDK Support 404 • SR-IOV and PCI Passthrough Support 414 • Setting

up VLAN-Aware VMs 430

10.5 Creating a Highly Available Router 438CVR and DVR High Available Routers 438 • Creating a High Availability

Router 439 • Test Router for High Availability 439

11 Managing the Dashboard 44111.1 Configuring the Dashboard Service 441

Dashboard Service and TLS in SUSE OpenStack Cloud 441

11.2 Changing the Dashboard Timeout Value 441How to Change the Dashboard Timeout Value 442

11.3 Creating a Load Balancer with the Dashboard 443

12 Managing Orchestration 44712.1 Configuring the Orchestration Service 447

ix Operations Guide CLM

12.2 Autoscaling using the Orchestration Service 449What is autoscaling? 449 • How does autoscaling work? 449 • Autoscaling

template example 450 • monasca Agent configuration options 450

12.3 Orchestration Service support for LBaaS v2 452Limitations 452 • More Information 453

13 Managing Monitoring, Logging, and UsageReporting 454

13.1 Monitoring 454Getting Started with Monitoring 454 • Configuring the Monitoring

Service 458 • Integrating HipChat, Slack, and JIRA 492 • Alarm

Metrics 498

13.2 Centralized Logging Service 584Getting Started with Centralized Logging Service 584 • Understanding the

Centralized Logging Service 586 • Accessing Log Data 595 • Managing

the Centralized Logging Feature 598 • Configuring Centralized

Logging 601 • Configuring Settings for Other Services 608 • Audit Logging

Overview 631 • Troubleshooting 641

13.3 Metering Service (ceilometer) Overview 641Metering Service New Functionality 641 • Understanding the

Metering Service Concepts 643 • Ceilometer Metering Available Meter

Types 645 • Configure the Ceilometer Metering Service 653 • Ceilometer

Metering Service Notifications 656 • Ceilometer Metering Setting

Role-based Access Control 665 • Ceilometer Metering Failover HA

Support 669 • Optimizing the Ceilometer Metering Service 671 • Metering

Service Samples 676

14 Managing Container as a Service (Magnum) 67814.1 Deploying a Kubernetes Cluster on Fedora Atomic 678

Prerequisites 678 • Creating the Cluster 678

14.2 Deploying a Kubernetes Cluster on CoreOS 683Prerequisites 683 • Creating the Cluster 684

x Operations Guide CLM

14.3 Deploying a Docker Swarm Cluster on Fedora Atomic 689Prerequisites 689 • Creating the Cluster 689

14.4 Deploying an Apache Mesos Cluster on Ubuntu 695Prerequisites 695 • Creating the Cluster 695

14.5 Creating a Magnum Cluster with the Dashboard 703Prerequisites 703 • Creating the Cluster Template 704 • Creating the

Cluster 706

15 System Maintenance 70815.1 Planned System Maintenance 708

Whole Cloud Maintenance 708 • Planned Control Plane

Maintenance 724 • Planned Compute Maintenance 730 • Planned

Network Maintenance 763 • Planned Storage

Maintenance 769 • Updating MariaDB with Galera 791

15.2 Unplanned System Maintenance 792Whole Cloud Recovery Procedures 792 • Recover Start-up

Processes 802 • Unplanned Control Plane Maintenance 804 • Unplanned

Compute Maintenance 820 • Unplanned Storage Maintenance 824

15.3 Cloud Lifecycle Manager Maintenance Update Procedure 826Performing the Update 827 • Summary of the Update Playbooks 830

15.4 Cloud Lifecycle Manager Program Temporary Fix (PTF)Deployment 832

16 Operations Console 83616.1 Using the Operations Console 836

Operations Console Overview 836 • Connecting to the Operations

Console 837 • Managing Compute Hosts 840 • Managing Swift

Performance 846 • Visualizing Data in Charts 854 • Getting Help with the

Operations Console 862

16.2 Alarm Definition 863Filter and Sort 863 • Create Alarm Definitions 864

xi Operations Guide CLM

16.3 Alarm Explorer 865Filter and Sort 865 • Alarm Table 865 • Notification Methods 866

16.4 Compute Hosts 867Filter and Sort 867

16.5 Compute Instances 868Search and Sort 868

16.6 Compute Summary 868Inventory Summary 869 • Capacity Summary 869 • Compute

Summary 869 • Appliances 869 • Block Storage Summary 870

16.7 Logging 871View Logging Interface 871

16.8 My Dashboard 872

16.9 Networking Alarm Summary 872Filter and Sort 872 • Alarm Table 872

16.10 Central Dashboard 873Central Dashboard 873 • New Alarms 874 • Alarm Summary 874

17 Backup and Restore 87617.1 Manual Backup Overview 876

17.2 Enabling Backups to a Remote Server 877Securing your SSH backup server 878 • General tips 878

17.3 Manual Backup and Restore Procedures 879Cloud Lifecycle Manager Data Backup 879 • MariaDB Database

Backup 883 • swift Ring Backup 886 • Audit Log Backup and

Restore 887

17.4 Full Disaster Recovery Test 888High Level View of the Recovery Process 888 • Description of the testing

environment 889 • Pre-Disaster testing 889 • Preparation of the test

backup server 890 • Perform Backups for disaster recovery test 892

xii Operations Guide CLM

18 Troubleshooting Issues 90418.1 General Troubleshooting 904

Alarm Resolution Procedures 904 • Support Resources 976

18.2 Control Plane Troubleshooting 977Understanding and Recovering RabbitMQ after Failure 977

18.3 Troubleshooting Compute service 984How can I reset the state of a compute instance? 985 • Troubleshooting nova-

consoleauth 985 • Enabling the migrate or resize functions in nova post-

installation when using encryption 987 • Compute (ESX) 990

18.4 Network Service Troubleshooting 991CVR HA - Split-brain result of failover of L3 agent when master comes back

up 991 • OVSvApp Loses Connectivity with vCenter 991 • Fail over a

plain CVR router because the node became unavailable: 992 • Trouble

setting maximum transmission units (MTU) 992 • Floating

IP on allowed_address_pair port with DVR-routed networks

allowed_address_pair 992 • OpenStack trac that must traverse VXLAN

tunnel dropped when using HPE 5930 switch 998 • Issue: PCI-PT virtual

machine gets stuck at boot 999

18.5 Troubleshooting the Image (glance) Service 1000Images Created in Horizon UI Get Stuck in a Queued State 1000

18.6 Storage Troubleshooting 1001Block Storage Troubleshooting 1001 • swift Storage Troubleshooting 1013

18.7 Monitoring, Logging, and Usage Reporting Troubleshooting 1025Troubleshooting Centralized Logging 1025 • Usage Reporting

Troubleshooting 1031

18.8 Orchestration Troubleshooting 1033Heat Troubleshooting 1033 • Troubleshooting Magnum Service 1035

18.9 Troubleshooting Tools 1039Retrieving the SOS Report 1039

xiii Operations Guide CLM

1 Operations Overview

A high-level overview of the processes related to operating a SUSE OpenStack Cloud 9 cloud.

1.1 What is a cloud operator?When we talk about a cloud operator it is important to understand the scope of the tasks andresponsibilities we are referring to. SUSE OpenStack Cloud defines a cloud operator as the personor group of people who will be administering the cloud infrastructure, which includes:

Monitoring the cloud infrastructure, resolving issues as they arise.

Managing hardware resources, adding/removing hardware due to capacity needs.

Repairing, and recovering if needed, any hardware issues.

Performing domain administration tasks, which involves creating and managing projects,users, and groups as well as setting and managing resource quotas.

1.2 Tools provided to operate your cloudSUSE OpenStack Cloud provides the following tools which are available to operate your cloud:

Operations Console

Often referred to as the Ops Console, you can use this console to view data about your cloudinfrastructure in a web-based graphical user interface (GUI) to make sure your cloud is operatingcorrectly. By logging on to the console, SUSE OpenStack Cloud administrators can manage datain the following ways:

Triage alarm notifications in the central dashboard

Monitor the environment by giving priority to alarms that take precedence

Manage compute nodes and easily use a form to create a new host

Refine the monitoring environment by creating new alarms to specify a combination ofmetrics, services, and hosts that match the triggers unique to an environment

Plan for future storage by tracking capacity over time to predict with some degree ofreliability the amount of additional storage needed

1 What is a cloud operator? SUSE OpenStack Cloud 9

Dashboard

Often referred to as horizon or the horizon dashboard, you can use this console to manageresources on a domain and project level in a web-based graphical user interface (GUI). Thefollowing are some of the typical operational tasks that you may perform using the dashboard:

Creating and managing projects, users, and groups within your domain.

Assigning roles to users and groups to manage access to resources.

Setting and updating resource quotas for the projects.

For more details, see the following page: Section 5.3, “Understanding Domains, Projects, Users,Groups, and Roles”

Command-line interface (CLI)

The OpenStack community has created a unified client, called the openstackclient (OSC), whichcombines the available commands in the various service-specific clients into one tool. Someservice-specific commands do not have OSC equivalents.

You will nd processes defined in our documentation that use these command-line tools. Thereis also a list of common cloud administration tasks which we have outlined which you can usethe command-line tools to do.

There are references throughout the SUSE OpenStack Cloud documentation to the HPESmart Storage Administrator (HPE SSA) CLI. HPE-specific binaries that are not basedon open source are distributed directly from and supported by HPE. To downloadand install the SSACLI utility, please refer to: https://support.hpe.com/hpsc/swd/public/de-tail?swItemId=MTX_3d16386b418a443388c18da82f

1.3 Daily tasks

Ensure your cloud is running correctly: SUSE OpenStack Cloud is deployed as a set ofhighly available services to minimize the impact of failures. That said, hardware and soft-ware systems can fail. Detection of failures early in the process will enable you to addressissues before they affect the broader system. SUSE OpenStack Cloud provides a monitor-ing solution, based on OpenStack’s monasca, which provides monitoring and metrics forall OpenStack components and much of the underlying system, including service status,performance metrics, compute node, and virtual machine status. Failures are exposed viathe Operations Console and/or alarm notifications. In the case where more detailed diag-

2 Daily tasks SUSE OpenStack Cloud 9

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_3d16386b418a443388c18da82fhttps://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_3d16386b418a443388c18da82f

nostics are required, you can use a centralized logging system based on the Elasticsearch,Logstash, and Kibana (ELK) stack. This provides the ability to search service logs to getdetailed information on behavior and errors.

Perform critical maintenance: To ensure your OpenStack installation is running correct-ly, provides the right access and functionality, and is secure, you should make ongoingadjustments to the environment. Examples of daily maintenance tasks include:

Add/remove projects and users. The frequency of this task depends on your policy.

Apply security patches (if released).

Run daily backups.

1.4 Weekly or monthly tasks

Do regular capacity planning: Your initial deployment will likely reflect the known nearto mid-term scale requirements, but at some point your needs will outgrow your initialdeployment’s capacity. You can expand SUSE OpenStack Cloud in a variety of ways, suchas by adding compute and storage capacity.

To manage your cloud’s capacity, begin by determining the load on the existing system. OpenS-tack is a set of relatively independent components and services, so there are multiple subsystemsthat can affect capacity. These include control plane nodes, compute nodes, object storage nodes,block storage nodes, and an image management system. At the most basic level, you shouldlook at the CPU used, RAM used, I/O load, and the disk space used relative to the amountsavailable. For compute nodes, you can also evaluate the allocation of resource to hosted virtualmachines. This information can be viewed in the Operations Console. You can pull historicalinformation from the monitoring service (OpenStack’s monasca) by using its client or API. Also,OpenStack provides you some ability to manage the hosted resource utilization by using quotasfor projects. You can track this usage over time to get your growth trend so that you can projectwhen you will need to add capacity.

3 Weekly or monthly tasks SUSE OpenStack Cloud 9

1.5 Semi-annual tasks

Perform upgrades: OpenStack releases new versions on a six-month cycle. In general,SUSE OpenStack Cloud will release new major versions annually with minor versions andmaintenance updates more often. Each new release consists of both new functionality andservices, as well as bug fixes for existing functionality.

NoteIf you are planning to upgrade, this is also an excellent time to evaluate your existingcapabilities, especially in terms of capacity (see Capacity Planning above).

1.6 TroubleshootingAs part of managing your cloud, you should be ready to troubleshoot issues, as needed. Thefollowing are some common troubleshooting scenarios and solutions:

How do I determine if my cloud is operating correctly now?: SUSE OpenStack Cloud pro-vides a monitoring solution based on OpenStack’s monasca service. This service provides mon-itoring and metrics for all OpenStack components, as well as much of the underlying system. Bydefault, SUSE OpenStack Cloud comes with a set of alarms that provide coverage of the primarysystems. In addition, you can define alarms based on threshold values for any metrics definedin the system. You can view alarm information in the Operations Console. You can also receiveor deliver this information to others by configuring email or other mechanisms. Alarms provideinformation about whether a component failed and is affecting the system, and also what con-dition triggered the alarm.

How do I troubleshoot and resolve performance issues for my cloud?: There are a varietyof factors that can affect the performance of a cloud system, such as the following:

Health of the control plane

Health of the hosting compute node and virtualization layer

Resource allocation on the compute node

4 Semi-annual tasks SUSE OpenStack Cloud 9

If your cloud users are experiencing performance issues on your cloud, use the following ap-proach:

1. View the compute summary page on the Operations Console to determine if any alarmshave been triggered.

2. Determine the hosting node of the virtual machine that is having issues.

3. On the compute hosts page, view the status and resource utilization of the compute nodeto determine if it has errors or is over-allocated.

4. On the compute instances page you can view the status of the VM along with its metrics.

How do I troubleshoot and resolve availability issues for my cloud?: If your cloud usersare experiencing availability issues, determine what your users are experiencing that indicatesto them the cloud is down. For example, can they not access the Dashboard service (horizon)console or APIs, indicating a problem with the control plane? Or are they having trouble access-ing resources? Console/API issues would indicate a problem with the control planes. Use theOperations Console to view the status of services to see if there is an issue. However, if it is anissue of accessing a virtual machine, then also search the consolidated logs that are available inthe ELK stack or errors related to the virtual machine and supporting networking.

1.7 Common QuestionsWhat skills do my cloud administrators need?

Your administrators should be experienced Linux admins. They should have experience in ap-plication management, as well as experience with Ansible. It is a plus if they have experiencewith Bash shell scripting and Python programming skills.

In addition, you will need skilled networking engineering sta to administer the cloud networkenvironment.

5 Common Questions SUSE OpenStack Cloud 9

2 Tutorials

This section contains tutorials for common tasks for your SUSE OpenStack Cloud 9 cloud.

2.1 SUSE OpenStack Cloud Quickstart Guide

2.1.1 Introduction

This document provides simplified instructions for installing and setting up a SUSE OpenStackCloud. Use this quickstart guide to build testing, demonstration, and lab-type environments.,rather than production installations. When you complete this quickstart process, you will havea fully functioning SUSE OpenStack Cloud demo environment.

NoteThese simplified instructions are intended for testing or demonstration. Instructions forproduction installations are in Book “Deployment Guide using Cloud Lifecycle Manager”.

2.1.2 Overview of components

The following are short descriptions of the components that SUSE OpenStack Cloud employswhen installing and deploying your cloud.

Ansible. Ansible is a powerful configuration management tool used by SUSE OpenStack Cloudto manage nearly all aspects of your cloud infrastructure. Most commands in this quickstartguide execute Ansible scripts, known as playbooks. You will run playbooks that install packages,edit configuration les, manage network settings, and take care of the general administrationtasks required to get your cloud up and running.

Get more information on Ansible at https://www.ansible.com/ .

Cobbler. Cobbler is another third-party tool used by SUSE OpenStack Cloud to deploy oper-ating systems across the physical servers that make up your cloud. Find more info at http://cobbler.github.io/ .

6 SUSE OpenStack Cloud Quickstart Guide SUSE OpenStack Cloud 9

https://www.ansible.com/http://cobbler.github.io/http://cobbler.github.io/

Git. Git is the version control system used to manage the configuration les that define yourcloud. Any changes made to your cloud configuration les must be committed to the locallyhosted git repository to take effect. Read more information on Git at https://git-scm.com/ .

2.1.3 Preparation

Successfully deploying a SUSE OpenStack Cloud environment is a large endeavor, but it is notcomplicated. For a successful deployment, you must put a number of components in place beforerolling out your cloud. Most importantly, a basic SUSE OpenStack Cloud requires the propernetwork infrastrucure. Because SUSE OpenStack Cloud segregates the network traffic of manyof its elements, if the necessary networks, routes, and firewall access rules are not in place,communication required for a successful deployment will not occur.

2.1.4 Getting Started

When your network infrastructure is in place, go ahead and set up the Cloud Lifecycle Manager.This is the server that will orchestrate the deployment of the rest of your cloud. It is also theserver you will run most of your deployment and management commands on.

Set up the Cloud Lifecycle Manager

1. Download the installation mediaObtain a copy of the SUSE OpenStack Cloud installation media, and make sure that it isaccessible by the server that you are installing it on. Your method of doing this may vary.For instance, some may choose to load the installation ISO on a USB drive and physicallyattach it to the server, while others may run the IPMI Remote Console and attach the ISOto a virtual disc drive.

2. Install the operating system

a. Boot your server, using the installation media as the boot source.

b. Choose "install" from the list of options and choose your preferred keyboard layout,location, language, and other settings.

c. Set the address, netmask, and gateway for the primary network interface.

d. Create a root user account.

7 Preparation SUSE OpenStack Cloud 9

https://git-scm.com/

Proceed with the OS installation. After the installation is complete and the server hasrebooted into the new OS, log in with the user account you created.

3. Configure the new server

a. SSH to your new server, and set a valid DNS nameserver in the /etc/resolv.confle.

b. Set the environment variable LC_ALL :

export LC_ALL=C

You now have a server running SUSE Linux Enterprise Server (SLES). The next step is toconfigure this machine as a Cloud Lifecycle Manager.

4. Configure the Cloud Lifecycle ManagerThe installation media you used to install the OS on the server also has the les that willconfigure your cloud. You need to mount this installation media on your new server inorder to use these les.

a. Using the URL that you obtained the SUSE OpenStack Cloud installation media from,run wget to download the ISO le to your server:

wget INSTALLATION_ISO_URL

b. Now mount the ISO in the /media/cdrom/ directory

sudo mount INSTALLATION_ISO /media/cdrom/

c. Unpack the tar le found in the /media/cdrom/ardana/ directory where you justmounted the ISO:

tar xvf /media/cdrom/ardana/ardana-x.x.x-x.tar

d. Now you will install and configure all the components needed to turn this server in-to a Cloud Lifecycle Manager. Run the ardana-init.bash script from the uncom-pressed tar le:

~/ardana-x.x.x/ardana-init.bash

8 Getting Started SUSE OpenStack Cloud 9

The ardana-init.bash script prompts you to enter an optional SSH passphrase.This passphrase protects the RSA key used to SSH to the other cloud nodes. This isan optional passphrase, and you can skip it by pressing Enter at the prompt.The ardana-init.bash script automatically installs and configures everythingneeded to set up this server as the lifecycle manager for your cloud.When the script has finished running, you can proceed to the next step, editing yourinput les.

5. Edit your input lesYour SUSE OpenStack Cloud input les are where you define your cloud infrastructureand how it runs. The input les define options such as which servers are included in yourcloud, the type of disks the servers use, and their network configuration. The input lesalso define which services your cloud will provide and use, the network architecture, andthe storage backends for your cloud.There are several example configurations, which you can nd on your Cloud LifecycleManager in the ~/openstack/examples/ directory.

a. The simplest way to set up your cloud is to copy the contents of one of these exam-ple configurations to your ~/openstack/mycloud/definition/ directory. You canthen edit the copied les and define your cloud.

cp -r ~/openstack/examples/CHOSEN_EXAMPLE/* ~/openstack/my_cloud/definition/

b. Edit the les in your ~/openstack/my_cloud/definition/ directory to define yourcloud.

6. Commit your changesWhen you finish editing the necessary input les, stage them, and then commit the changesto the local Git repository:

cd ~/openstack/ardana/ansiblegit add -Agit commit -m "My commit message"

7. Image your serversNow that you have finished editing your input les, you can deploy the configuration tothe servers that will comprise your cloud.


a. Image the servers. You will install the SLES operating system across all the serversin your cloud, using Ansible playbooks to trigger the process.

b. The following playbook confirms that your servers are accessible over their IPMIports, which is a prerequisite for the imaging process:

ansible-playbook -i hosts/localhost bm-power-status.yml

c. Now validate that your cloud configuration les have proper YAML syntax by run-ning the config-processor-run.yml playbook:

ansible-playbook -i hosts/localhost config-processor-run.yml

If you receive an error when running the preceeding playbook, one or more of yourconfiguration les has an issue. Refer to the output of the Ansible playbook, and lookfor clues in the Ansible log le, found at ~/.ansible/ansible.log .

d. The next step is to prepare your imaging system, Cobbler, to deploy operating sys-tems to all your cloud nodes:

ansible-playbook -i hosts/localhost cobbler-deploy.yml

e. Now you can image your cloud nodes. You will use an Ansible playbook to triggerCobbler to deploy operating systems to all the nodes you specified in your input les:

ansible-playbook -i hosts/localhost bm-reimage.yml

The bm-reimage.yml playbook performs the following operations:

1. Powers down the servers.

2. Sets the servers to boot from a network interface.

3. Powers on the servers and performs a PXE OS installation.

4. Waits for the servers to power themselves down as part of a successful OSinstallation. This can take some time.

5. Sets the servers to boot from their local hard disks and powers on the servers.

6. Waits for the SSH service to start on the servers and verifies that they have theexpected host-key signature.


8. Deploy your cloudNow that your servers are running the SLES operating system, it is time to configure themfor the roles they will play in your new cloud.

a. Prepare the Cloud Lifecycle Manager to deploy your cloud configuration to all thenodes:

ansible-playbook -i hosts/localhost ready-deployment.yml

NOTE: The preceding playbook creates a new directory, ~/scratch/ansible/next/ardana/ansible/ , from which you will run many of the following commands.

b. (Optional) If you are reusing servers or disks to run your cloud, you can wipe thedisks of your newly imaged servers by running the wipe_disks.yml playbook:

cd ~/scratch/ansible/next/ardana/ansible/ansible-playbook -i hosts/verb_hosts wipe_disks.yml

The wipe_disks.yml playbook removes any existing data from the drives on yournew servers. This can be helpful if you are reusing servers or disks. This action willnot affect the OS partitions on the servers.

NoteThe wipe_disks.yml playbook is only meant to be run on systems immedi-ately after running bm-reimage.yml . If used for any other case, it may notwipe all of the expected partitions. For example, if site.yml fails, you can-not start fresh by running wipe_disks.yml . You must bm-reimage the noderst and then run wipe_disks .

c. Now it is time to deploy your cloud. Do this by running the site.yml playbook,which pushes the configuration you defined in the input les out to all the serversthat will host your cloud.

cd ~/scratch/ansible/next/ardana/ansible/ansible-playbook -i hosts/verb_hosts site.yml


The site.yml playbook installs packages, starts services, configures network inter-face settings, sets iptables firewall rules, and more. Upon successful completion ofthis playbook, your SUSE OpenStack Cloud will be in place and in a running state.This playbook can take up to six hours to complete.

9. SSH to your nodesNow that you have successfully run site.yml , your cloud will be up and running. Youcan verify connectivity to your nodes by connecting to each one by using SSH. You cannd the IP addresses of your nodes by viewing the /etc/hosts le.For security reasons, you can only SSH to your nodes from the Cloud Lifecycle Manager.SSH connections from any machine other than the Cloud Lifecycle Manager will be refusedby the nodes.From the Cloud Lifecycle Manager, SSH to your nodes:

ssh

Also note that SSH is limited to your cloud's management network. Each node has anaddress on the management network, and you can nd this address by reading the /etc/hosts or server_info.yml le.

2.2 Installing the Command-Line ClientsDuring the installation, by default, the suite of OpenStack command-line tools are installedon the Cloud Lifecycle Manager and the control plane in your environment. You can learnmore about these in the OpenStack documentation here: OpenStackClient (https://docs.open-stack.org/python-openstackclient/latest/) .

If you wish to install the command-line interfaces on other nodes in your environment, thereare two methods you can use to do so that we describe below.

2.2.1 Installing the CLI tools using the input model

During the initial install phase of your cloud you can edit your input model to request that thecommand-line clients be installed on any of the node clusters in your environment. To do so,follow these steps:

1. Log in to the Cloud Lifecycle Manager.

12 Installing the Command-Line Clients SUSE OpenStack Cloud 9

https://docs.openstack.org/python-openstackclient/latest/https://docs.openstack.org/python-openstackclient/latest/

2. Edit your control_plane.yml le. Full path:

~/openstack/my_cloud/definition/data/control_plane.yml

3. In this le you will see a list of service-components to be installed on each of yourclusters. These clusters will be divided per role, with your controller node cluster likelycoming at the beginning. Here you will see a list of each of the clients that can be installed.These include:

keystone-clientglance-clientcinder-clientnova-clientneutron-clientswift-clientheat-clientopenstack-clientmonasca-clientbarbican-clientdesignate-client

4. For each client you want to install, specify the name under the service-componentssection for the cluster you want to install it on.So, for example, if you would like to install the nova and neutron clients on your Computenode cluster, you can do so by adding the nova-client and neutron-client services,like this:

resources: - name: compute resource-prefix: comp server-role: COMPUTE-ROLE allocation-policy: any min-count: 0 service-components: - ntp-client - nova-compute - nova-compute-kvm - neutron-l3-agent - neutron-metadata-agent - neutron-openvswitch-agent - nova-client - neutron-client

13 Installing the CLI tools using the input model SUSE OpenStack Cloud 9

NoteThis example uses the entry-scale-kvm sample le. Your model may be differentso use this as a guide but do not copy and paste the contents of this example intoyour input model.

5. Commit your configuration to the local git repo, as follows:

cd ~/openstack/ardana/ansiblegit add -Agit commit -m "My config or other commit message"

6. Continue with the rest of your installation.

2.2.2 Installing the CLI tools using Ansible

At any point after your initial installation you can install the command-line clients on any ofthe nodes in your environment. To do so, follow these steps:

1. Log in to the Cloud Lifecycle Manager.

2. Obtain the hostname for the nodes you want to install the clients on by looking in yourhosts le:

cat /etc/hosts

3. Install the clients using this playbook, specifying your hostnames using commas:

cd ~/scratch/ansible/next/ardana/ansibleansible-playbook -i hosts/verb_hosts -e "install_package=" client-deploy.yml -e "install_hosts="

So, for example, if you would like to install the novaClient on two of your Compute nodeswith hostnames ardana-cp1-comp0001-mgmt and ardana-cp1-comp0002-mgmt you canuse this syntax:

cd ~/scratch/ansible/next/ardana/ansible

14 Installing the CLI tools using Ansible SUSE OpenStack Cloud 9

ansible-playbook -i hosts/verb_hosts -e "install_package=novaclient" client-deploy.yml -e "install_hosts=ardana-cp1-comp0001-mgmt,ardana-cp1-comp0002-mgmt"

4. Once the playbook completes successfully, you should be able to SSH to those nodes and,using the proper credentials, authenticate and use the command-line interfaces you haveinstalled.

2.3 Cloud Admin Actions with the Command LineCloud admins can use the command line tools to perform domain admin tasks such as user andproject administration.

2.3.1 Creating Additional Cloud Admins

You can create additional Cloud Admins to help with the administration of your cloud.

keystone identity service query and administration tasks can be performed using the OpenStackcommand line utility. The utility is installed by the Cloud Lifecycle Manager onto the CloudLifecycle Manager.

Notekeystone administration tasks should be performed by an admin user with a token scopedto the default domain via the keystone v3 identity API. These settings are preconfiguredin the le ~/keystone.osrc . By default, keystone.osrc is configured with the adminendpoint of keystone. If the admin endpoint is not accessible from your network, changeOS_AUTH_URL to point to the public endpoint.

2.3.2 Command Line Examples

For a full list of OpenStackClient commands, see OpenStackClient Command List (http://doc-s.openstack.org/developer/python-openstackclient/command-list.html) .

Sourcing the keystone Administration Credentials

15 Cloud Admin Actions with the Command Line SUSE OpenStack Cloud 9

http://docs.openstack.org/developer/python-openstackclient/command-list.htmlhttp://docs.openstack.org/developer/python-openstackclient/command-list.html

You can set the environment variables needed for identity administration by sourcing the key-stone.osrc le created by the lifecycle manager:

source ~/keystone.osrc

List users in the default domain

These users are created by the Cloud Lifecycle Manager in the MySQL back end:

openstack user list

Example output:

$ openstack user list+----------------------------------+------------------+| ID | Name |+----------------------------------+------------------+| 155b68eda9634725a1d32c5025b91919 | heat || 303375d5e44d48f298685db7e6a4efce | octavia || 40099e245a394e7f8bb2aa91243168ee | logging || 452596adbf4d49a28cb3768d20a56e38 | admin || 76971c3ad2274820ad5347d46d7560ec | designate || 7b2dc0b5bb8e4ffb92fc338f3fa02bf3 | hlm_backup || 86d345c960e34c9189519548fe13a594 | barbican || 8e7027ab438c4920b5853d52f1e08a22 | nova_monasca || 9c57dfff57e2400190ab04955e7d82a0 | barbican_service || a3f99bcc71b242a1bf79dbc9024eec77 | nova || aeeb56fc4c4f40e0a6a938761f7b154a | glance-check || af1ef292a8bb46d9a1167db4da48ac65 | cinder || af3000158c6d4d3d9257462c9cc68dda | demo || b41a7d0cb1264d949614dc66f6449870 | swift || b78a2b17336b43368fb15fea5ed089e9 | cinderinternal || bae1718dee2d47e6a75cd6196fb940bd | monasca || d4b9b32f660943668c9f5963f1ff43f9 | ceilometer || d7bef811fb7e4d8282f19fb3ee5089e9 | swift-monitor || e22bbb2be91342fd9afa20baad4cd490 | neutron || ec0ad2418a644e6b995d8af3eb5ff195 | glance || ef16c37ec7a648338eaf53c029d6e904 | swift-dispersion || ef1a6daccb6f4694a27a1c41cc5e7a31 | glance-swift || fed3a599b0864f5b80420c9e387b4901 | monasca-agent |+----------------------------------+------------------+

List domains created by the installation process:

openstack domain list

Example output:

$ openstack domain list

16 Command Line Examples SUSE OpenStack Cloud 9

+----------------------------------+---------+---------+----------------------------------------------------------------------+| ID | Name | Enabled | Description |+----------------------------------+---------+---------+----------------------------------------------------------------------+| 6740dbf7465a4108a36d6476fc967dbd | heat | True | Owns users and projects created by heat || default | Default | True | Owns users and tenants (i.e. projects) available on Identity API v2. |+----------------------------------+---------+---------+----------------------------------------------------------------------+

List the roles:

openstack role list

Example output:

$ openstack role list+----------------------------------+---------------------------+| ID | Name |+----------------------------------+---------------------------+| 0be3da26cd3f4cd38d490b4f1a8b0c03 | designate_admin || 13ce16e4e714473285824df8188ee7c0 | monasca-agent || 160f25204add485890bc95a6065b9954 | key-manager:service-admin || 27755430b38c411c9ef07f1b78b5ebd7 | monitor || 2b8eb0a261344fbb8b6b3d5934745fe1 | key-manager:observer || 345f1ec5ab3b4206a7bffdeb5318bd32 | admin || 49ba3b42696841cea5da8398d0a5d68e | nova_admin || 5129400d4f934d4fbfc2c3dd608b41d9 | ResellerAdmin || 60bc2c44f8c7460a9786232a444b56a5 | neutron_admin || 654bf409c3c94aab8f929e9e82048612 | cinder_admin || 854e542baa144240bfc761cdb5fe0c07 | monitoring-delegate || 8946dbdfa3d346b2aa36fa5941b43643 | key-manager:auditor || 901453d9a4934610ad0d56434d9276b4 | key-manager:admin || 9bc90d1121544e60a39adbfe624a46bc | monasca-user || 9fe2a84a3e7443ae868d1009d6ab4521 | service || 9fe2ff9ee4384b1894a90878d3e92bab | member || a24d4e0a5de14bffbe166bfd68b36e6a | swiftoperator || ae088fcbf579425580ee4593bfa680e5 | heat_stack_user || bfba56b2562942e5a2e09b7ed939f01b | keystoneAdmin || c05f54cf4bb34c7cb3a4b2b46c2a448b | glance_admin || fe010be5c57240db8f559e0114a380c1 | key-manager:creator |+----------------------------------+---------------------------+


List admin user role assignment within default domain:

openstack role assignment list --user admin --domain default

Example output:

# This indicates that the admin user is assigned the admin role within the default domainardana > openstack role assignment list --user admin --domain default+----------------------------------+----------------------------------+-------+---------+---------+| Role | User | Group | Project | Domain |+----------------------------------+----------------------------------+-------+---------+---------+| b398322103504546a070d607d02618ad | fed1c038d9e64392890b6b44c38f5bbb | | | default |+----------------------------------+----------------------------------+-------+---------+---------+

Create a new user in default domain:

openstack user create --domain default --password-prompt --email --description --enable

Example output showing the creation of a user named testuser with email address [email protected] and a description of Test User :

ardana > openstack user create --domain default --password-prompt --email [email protected] --description "Test User" --enable testuserUser Password:Repeat User Password:+-------------+----------------------------------+| Field | Value |+-------------+----------------------------------+| description | Test User || domain_id | default || email | [email protected] || enabled | True || id | 8aad69acacf0457e9690abf8c557754b || name | testuser |+-------------+----------------------------------+

Assign admin role for testuser within the default domain:

openstack role add admin --user --domain defaultopenstack role assignment list --user --domain default


Example output:

# Just for demonstration purposes - do not do this in a production environment!ardana > openstack role add admin --user testuser --domain defaultardana > openstack role assignment list --user testuser --domain default+----------------------------------+----------------------------------+-------+---------+---------+| Role | User | Group | Project | Domain |+----------------------------------+----------------------------------+-------+---------+---------+| b398322103504546a070d607d02618ad | 8aad69acacf0457e9690abf8c557754b | | | default |+----------------------------------+----------------------------------+-------+---------+---------+

2.3.3 Assigning the default service admin roles

The following examples illustrate how you can assign each of the new service admin roles toa user.

Assigning the glance_admin role

A user must have the role of admin in order to assign the glance_admin role. To assign the role,you will set the environment variables needed for the identity service administrator.

1. First, source the identity service credentials:


2. You can add the glance_admin role to a user on a project with this command:

openstack role add --user --project glance_admin

Example, showing a user named testuser being granted the glance_admin role in thetest_project project:

openstack role add --user testuser --project test_project glance_admin

3. You can confirm the role assignment by listing out the roles:

openstack role assignment list --user

19 Assigning the default service admin roles SUSE OpenStack Cloud 9

Example output:

ardana > openstack role assignment list --user testuser+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+| Role | User | Group | Project | Domain | Inherited |+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+| 46ba80078bc64853b051c964db918816 | 8bcfe10101964e0c8ebc4de391f3e345 | | 0ebbf7640d7948d2a17ac08bbbf0ca5b | | False |+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+

4. Note that only the role ID is displayed. To get the role name, execute the following:

openstack role show

Example output:

ardana > openstack role show 46ba80078bc64853b051c964db918816+-------+----------------------------------+| Field | Value |+-------+----------------------------------+| id | 46ba80078bc64853b051c964db918816 || name | glance_admin |+-------+----------------------------------+

5. To demonstrate that the user has glance admin privileges, authenticate with those usercreds and then upload and publish an image. Only a user with an admin role or glance_ad-min can publish an image.

a. The easiest way to do this will be to make a copy of the service.osrc le and editit with your user credentials. You can do that with this command:

cp ~/service.osrc ~/user.osrc

b. Using your preferred editor, edit the user.osrc le and replace the values for thefollowing entries to match your user credentials:

export OS_USERNAME=


export OS_PASSWORD=

c. You will also need to edit the following lines for your environment:

## Change these values from 'unset' to 'export'export OS_PROJECT_NAME=export OS_PROJECT_DOMAIN_NAME=Default

Here is an example output:

unset OS_DOMAIN_NAMEexport OS_IDENTITY_API_VERSION=3export OS_AUTH_VERSION=3export OS_PROJECT_NAME=test_projectexport OS_PROJECT_DOMAIN_NAME=Defaultexport OS_USERNAME=testuserexport OS_USER_DOMAIN_NAME=Defaultexport OS_PASSWORD=testuserexport OS_AUTH_URL=http://192.168.245.9:35357/v3export OS_ENDPOINT_TYPE=internalURL# OpenstackClient uses OS_INTERFACE instead of OS_ENDPOINTexport OS_INTERFACE=internalexport OS_CACERT=/etc/ssl/certs/ca-certificates.crt

6. Source the environment variables for your user:

source ~/user.osrc

7. Upload an image and publicize it:

openstack image create --name "upload me" --visibility public --container-format bare --disk-format qcow2 --file uploadme.txt

Example output:

+------------------+--------------------------------------+| Property | Value |+------------------+--------------------------------------+| checksum | dd75c3b840a16570088ef12f6415dd15 || container_format | bare || created_at | 2016-01-06T23:31:27Z || disk_format | qcow2 || id | cf1490f4-1eb1-477c-92e8-15ebbe91da03 || min_disk | 0 || min_ram | 0 || name | upload me |


| owner | bd24897932074780a20b780c4dde34c7 || protected | False || size | 10 || status | active || tags | [] || updated_at | 2016-01-06T23:31:31Z || virtual_size | None || visibility | public |+------------------+--------------------------------------+

NoteYou can use the command openstack help image create to get the full syntaxfor this command.

Assigning the nova_admin role

A user must have the role of admin in order to assign the nova_admin role. To assign the role,you will set the environment variables needed for the identity service administrator.

1. First, source the identity service credentials:


2. You can add the glance_admin role to a user on a project with this command:

openstack role add --user --project nova_admin

Example, showing a user named testuser being granted the glance_admin role in thetest_project project:

openstack role add --user testuser --project test_project nova_admin

3. You can confirm the role assignment by listing out the roles:

openstack role assignment list --user

Example output:

ardana > openstack role assignment list --user testuser+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+| Role | User | Group | Project | Domain | Inherited |


+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+| 8cdb02bab38347f3b65753099f3ab73c | 8bcfe10101964e0c8ebc4de391f3e345 | | 0ebbf7640d7948d2a17ac08bbbf0ca5b | | False |+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+

4. Note that only the role ID is displayed. To get the role name, execute the following:

openstack role show

Example output:

ardana > openstack role show 8cdb02bab38347f3b65753099f3ab73c+-------+----------------------------------+| Field | Value |+-------+----------------------------------+| id | 8cdb02bab38347f3b65753099f3ab73c || name | nova_admin |+-------+----------------------------------+

5. To demonstrate that the user has nova admin privileges, authenticate with those user credsand then upload and publish an image. Only a user with an admin role or glance_admincan publish an image.

a. The easiest way to do this will be to make a copy of the service.osrc le and editit with your user credentials. You can do that with this command:

cp ~/service.osrc ~/user.osrc

b. Using your preferred editor, edit the user.osrc le and replace the values for thefollowing entries to match your user credentials:

export OS_USERNAME=export OS_PASSWORD=

c. You will also need to edit the following lines for your environment:

## Change these values from 'unset' to 'export'export OS_PROJECT_NAME=export OS_PROJECT_DOMAIN_NAME=Default


Here is an example output:

unset OS_DOMAIN_NAMEexport OS_IDENTITY_API_VERSION=3export OS_AUTH_VERSION=3export OS_PROJECT_NAME=test_projectexport OS_PROJECT_DOMAIN_NAME=Defaultexport OS_USERNAME=testuserexport OS_USER_DOMAIN_NAME=Defaultexport OS_PASSWORD=testuserexport OS_AUTH_URL=http://192.168.245.9:35357/v3export OS_ENDPOINT_TYPE=internalURL# OpenstackClient uses OS_INTERFACE instead of OS_ENDPOINTexport OS_INTERFACE=internalexport OS_CACERT=/etc/ssl/certs/ca-certificates.crt

6. Source the environment variables for your user:

source ~/user.osrc

7. List all of the virtual machines in the project specified in user.osrc:

openstack server list

Example output showing no virtual machines, because there are no virtual machines cre-ated on the project specified in the user.osrc le:

+--------------------------------------+-------------------------------------------------------+--------+-----------------------------------------------------------------+| ID | Name | Status | Networks |+--------------------------------------+-------------------------------------------------------+--------+-----------------------------------------------------------------++--------------------------------------+-------------------------------------------------------+--------+-----------------------------------------------------------------+


8. For this demonstration, we do have a virtual machine associated with a different projectand because your user has nova_admin permissions, you can view those virtual machinesusing a slightly different command:

openstack server list --all-projects

Example output, now showing a virtual machine:

ardana > openstack server list --all-projects+--------------------------------------+-------------------------------------------------------+--------+-----------------------------------------------------------------+| ID | Name | Status | Networks |+--------------------------------------+-------------------------------------------------------+--------+-----------------------------------------------------------------+| da4f46e2-4432-411b-82f7-71ab546f91f3 | testvml | ACTIVE | |+--------------------------------------+-------------------------------------------------------+--------+-----------------------------------------------------------------+

9. You can also now delete virtual machines in other projects by using the --all-tenantsswitch:

openstack server delete --all-projects

Example, showing us deleting the instance in the previous step:

openstack server delete --all-projects da4f46e2-4432-411b-82f7-71ab546f91f3

10. You can get a full list of available commands by using this:

openstack -h

You can perform the same steps as above for the neutron and cinder service admin roles:

neutron_admincinder_admin


2.3.4 Customize policy.json on the Cloud Lifecycle Manager

One way to deploy policy.json for a service is by going to each of the target nodes and makingchanges there. This is not necessary anymore. This process has been streamlined and policy.jsonles can be edited on the Cloud Lifecycle Manager and then deployed to nodes. Please exercisecaution when modifying policy.json les. It is best to validate the changes in a non-productionenvironment before rolling out policy.json changes into production. It is not recommended thatyou make policy.json changes without a way to validate the desired policy behavior. Updatedpolicy.json les can be deployed using the appropriate -reconfigure.ymlplaybook.

2.3.5 Roles

Service roles represent the functionality used to implement the OpenStack role based accesscontrol (RBAC) model. This is used to manage access to each OpenStack service. Roles are namedand assigned per user or group for each project by the identity service. Role definition andpolicy enforcement are defined outside of the identity service independently by each OpenStackservice.

The token generated by the identity service for each user authentication contains the role(s)assigned to that user for a particular project. When a user attempts to access a specific OpenStackservice, the role is parsed by the service, compared to the service-specific policy le, and thengranted the resource access defined for that role by the service policy le.

Each service has its own service policy le with the /etc/[SERVICE_CODENAME]/policy.jsonle name format where [SERVICE_CODENAME] represents a specific OpenStack service name.For example, the OpenStack nova service would have a policy le called /etc/nova/poli-cy.json .

Service policy les can be modified and deployed to control nodes from the Cloud LifecycleManager. Administrators are advised to validate policy changes before checking in the changesto the site branch of the local git repository before rolling the changes into production. Do notmake changes to policy les without having a way to validate them.

The policy les are located at the following site branch directory on the Cloud Lifecycle Manager.

~/openstack/ardana/ansible/roles/

26 Customize policy.json on the Cloud Lifecycle Manager SUSE OpenStack Cloud 9

For test and validation, policy les can be modified in a non-production environment from the~/scratch/ directory. For a specific policy le, run a search for policy.json . To deploypolicy changes for a service, run the service specific reconfiguration playbook (for example,nova-reconfigure.yml ). For a complete list of reconfiguration playbooks, change directoriesto ~/scratch/ansible/next/ardana/ansible and run this command:

ls –l | grep reconfigure

NoteComments added to any *.j2 les (including templates) must follow proper commentsyntax. Otherwise you may see errors when running the config-processor or any of theservice playbooks.

2.4 Log Management and Integration

2.4.1 Overview

SUSE OpenStack Cloud uses the ELK (Elasticsearch, Logstash, Kibana) stack for log managementacross the entire cloud infrastructure. This configuration facilitates simple administration aswell as integration with third-party tools. This tutorial covers how to forward your logs to athird-party tool or service, and how to access and search the Elasticsearch log stores throughAPI endpoints.

2.4.2 The ELK stack

The ELK logging stack consists of the Elasticsearch, Logstash, and Kibana elements.

27 Log Management and Integration SUSE OpenStack Cloud 9

Elasticsearch. Elasticsearch is the storage and indexing component of the ELK stack. Itstores and indexes the data received from Logstash. Indexing makes your log data search-able by tools designed for querying and analyzing massive sets of data. You can query theElasticsearch datasets from the built-in Kibana console, a third-party data analysis tool, orthrough the Elasticsearch API (covered later).

Logstash. Logstash reads the log data from the services running on your servers, and thenaggregates and ships that data to a storage location. By default, Logstash sends the datato the Elasticsearch indexes, but it can also be configured to send data to other storageand indexing tools such as Splunk.

Kibana. Kibana provides a simple and easy-to-use method for searching, analyzing, andvisualizing the log data stored in the Elasticsearch indexes. You can customize the Kibanaconsole to provide graphs, charts, and other visualizations of your log data.

2.4.3 Using the Elasticsearch API

You can query the Elasticsearch indexes through various language-specific APIs, as well as di-rectly over the IP address and port that Elasticsearch exposes on your implementation. By de-fault, Elasticsearch presents from localhost, port 9200. You can run queries directly from a ter-minal using curl . For example:

ardana > curl -XGET 'http://localhost:9200/_search?q=tag:yourSearchTag'

The preceding command searches all indexes for all data with the "yourSearchTag" tag.

You can also use the Elasticsearch API from outside the logging node. This method connectsover the Kibana VIP address, port 5601, using basic http authentication. For example, you canuse the following command to perform the same search as the preceding search:

curl -u kibana: kibana_vip:5601/_search?q=tag:yourSearchTag

You can further refine your search to a specific index of data, in this case the "elasticsearch"index:

ardana > curl -XGET 'http://localhost:9200/elasticsearch/_search?q=tag:yourSearchTag'

The search API is RESTful, so responses are provided in JSON format. Here's a sample (thoughempty) response:

{ "took":13,

28 Using the Elasticsearch API SUSE OpenStack Cloud 9

"timed_out":false, "_shards":{ "total":45, "successful":45, "failed":0 }, "hits":{ "total":0, "max_score":null, "hits":[] }}

2.4.4 For More Information

You can nd more detailed Elasticsearch API documentation at https://www.elastic.co/guide/en/elasticsearch/reference/current/search.html .

Review the Elasticsearch Python API documentation at the following sources: http://elastic-search-py.readthedocs.io/en/master/api.html

Read the Elasticsearch Java API documentation at https://www.elastic.co/guide/en/elastic-search/client/java-api/current/index.html .

2.4.5 Forwarding your logs

You can configure Logstash to ship your logs to an outside storage and indexing system, such asSplunk. Setting up this configuration is as simple as editing a few configuration les, and thenrunning the Ansible playbooks that implement the changes. Here are the steps.

1. Begin by logging in to the Cloud Lifecycle Manager.

2. Verify that the logging system is up and running:

ardana > cd ~/scratch/ansible/next/ardana/ansibleardana > ansible-playbook -i hosts/verb_hosts logging-status.yml

When the preceding playbook completes without error, proceed to the next step.

3. Edit the Logstash configuration le, found at the following location:

~/openstack/ardana/ansible/roles/logging-server/templates/logstash.conf.j2

29 For More Information SUSE OpenStack Cloud 9

https://www.elastic.co/guide/en/elasticsearch/reference/current/search.htmlhttps://www.elastic.co/guide/en/elasticsearch/reference/current/search.htmlhttp://elasticsearch-py.readthedocs.io/en/master/api.htmlhttp://elasticsearch-py.readthedocs.io/en/master/api.htmlhttps://www.elastic.co/guide/en/elasticsearch/client/java-api/current/index.htmlhttps://www.elastic.co/guide/en/elasticsearch/client/java-api/current/index.html

Near the end of the Logstash configuration le, you will nd a section for configuringLogstash output destinations. The following example demonstrates the changes necessaryto forward your logs to an outside server (changes in bold). The configuration block setsup a TCP connection to the destination server's IP address over port 5514.

# Logstash outputs output { # Configure Elasticsearch output # http://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html elasticsearch { index => "${[@metadata][es_index]"} hosts => ["{{ elasticsearch_http_host }}:{{ elasticsearch_http_port }}"] flush_size => {{ logstash_flush_size }} idle_flush_time => 5 workers => {{ logstash_threads }} } # Forward Logs to Splunk on TCP port 5514 which matches the one specified in Splunk Web UI. tcp { mode => "client" host => "" port => 5514 } }

Logstash can forward log data to multiple sources, so there is no need to remove or alterthe Elasticsearch section in the preceding le. However, if you choose to stop forwardingyour log data to Elasticsearch, you can do so by removing the related section in this le,and then continue with the following steps.

4. Commit your changes to the local git repository:

ardana > cd ~/openstack/ardana/ansibleardana > git add -Aardana > git commit -m "Your commit message"

5. Run the configuration processor to check the status of all configuration les:

ardana > ansible-playbook -i hosts/localhost config-processor-run.yml

30 Forwarding your logs SUSE OpenStack Cloud 9

6. Run the ready-deployment playbook:

ardana > ansible-playbook -i hosts/localhost ready-deployment.yml

7. Implement the changes to the Logstash configuration le:

ardana > cd ~/scratch/ansible/next/ardana/ansibleardana > ansible-playbook -i hosts/verb_hosts kronos-server-configure.yml

Configuring the receiving service will vary from product to product. Consult the documentationfor your particular product for instructions on how to set it up to receive log les from Logstash.

2.5 Integrating Your Logs with Splunk

2.5.1 Integrating with Splunk

The SUSE OpenStack Cloud 9 logging solution provides a flexible and extensible frameworkto centralize the collection and processing of logs from all nodes in your cloud. The logs areshipped to a highly available and fault-tolerant cluster where they are transformed and stored forbetter searching and reporting. The SUSE OpenStack Cloud 9 logging solution uses the ELK stack(Elasticsearch, Logstash and Kibana) as a production-grade implementation and can supportother storage and indexing technologies.

You can configure Logstash, the service that aggregates and forwards the logs to a searchableindex, to send the logs to a third-party target, such as Splunk.

For how to integrate the SUSE OpenStack Cloud 9 centralized logging solution with Splunk,including the steps to set up and forward logs, please refer to Section 4.1, “Splunk Integration”.

2.6 Integrating SUSE OpenStack Cloud with an LDAPSystemYou can configure your SUSE OpenStack Cloud cloud to work with an outside user authentica-tion source such as Active Directory or OpenLDAP. keystone, the SUSE OpenStack Cloud iden-tity service, functions as the rst stop for any user authorization/authentication requests. key-stone can also function as a proxy for user account authentication, passing along authenticationand authorization requests to any LDAP-enabled system that has been configured as an outside

31 Integrating Your Logs with Splunk SUSE OpenStack Cloud 9

source. This type of integration lets you use an existing user-management system such as Ac-tive Directory and its powerful group-based organization features as a source for permissionsin SUSE OpenStack Cloud.

Upon successful completion of this tutorial, your cloud will refer user authentication requeststo an outside LDAP-enabled directory system, such as Microsoft Active Directory or OpenLDAP.

2.6.1 Configure your LDAP source

To configure your SUSE OpenStack Cloud cloud to use an outside user-management source,perform the following steps:

1. Make sure that the LDAP-enabled system you plan to integrate with is up and running andaccessible over the necessary ports from your cloud management network.

2. Edit the /var/lib/ardana/openstack/my_cloud/config/keystone/keystone.con-f.j2 le and set the following options:

domain_specific_drivers_enabled = Truedomain_configurations_from_database = False

3. Create a YAML le in the /var/lib/ardana/openstack/my_cloud/config/keystone/directory that defines your LDAP connection. You can make a copy of the sample key-stone-LDAP configuration le, and then edit that le with the details of your LDAP con-nection.The following example copies the keystone_configure_ldap_sample.yml le andnames the new le keystone_configure_ldap_my.yml :

ardana > cp /var/lib/ardana/openstack/my_cloud/config/keystone/keystone_configure_ldap_sample.yml \ /var/lib/ardana/openstack/my_cloud/config/keystone/keystone_configure_ldap_my.yml

4. Edit the new le to define the connection to your LDAP source. This guide does not providecomprehensive information on all aspects of the keystone_configure_ldap.yml le.Find a complete list of keystone/LDAP configuration le options at: https://github.com/openstack/keystone/tree/stable/rocky/etc

The following le illustrates an example keystone configuration that is customized for anActive Directory connection.

keystone_domainldap_conf:

32 Configure your LDAP source SUSE OpenStack Cloud 9

https://github.com/openstack/keystone/tree/stable/rocky/etchttps://github.com/openstack/keystone/tree/stable/rocky/etc

# CA certificates file content. # Certificates are stored in Base64 PEM format. This may be entire LDAP server # certificate (in case of self-signed certificates), certificate of authority # which issued LDAP server certificate, or a full certificate chain (Root CA # certificate, intermediate CA certificate(s), issuer certificate). # cert_settings: cacert: | -----BEGIN CERTIFICATE-----

certificate appears here

-----END CERTIFICATE-----

# A domain will be created in MariaDB with this name, and associated with ldap back end. # Installer will also generate a config file named /etc/keystone/domains/keystone..conf # domain_settings: name: ad description: Dedicated domain for ad users

conf_settings: identity: driver: ldap

# For a full list and description of ldap configuration options, please refer to # http://docs.openstack.org/liberty/config-reference/content/keystone-configuration-file.html. # # Please note: # 1. LDAP configuration is read-only. Configuration which performs write operations (i.e. creates users, groups, etc) # is not supported at the moment. # 2. LDAP is only supported for identity operations (reading users and groups from LDAP). Assignment # operations with LDAP (i.e. managing roles, projects) are not supported. # 3. LDAP is configured as non-default domain. Configuring LDAP as a default domain is not supported. #

ldap: url: ldap://YOUR_COMPANY_AD_URL suffix: YOUR_COMPANY_DC


query_scope: sub user_tree_dn: CN=Users,YOUR_COMPANY_DC user : CN=admin,CN=Users,YOUR_COMPANY_DC password: REDACTED user_objectclass: user user_id_attribute: cn user_name_attribute: cn group_tree_dn: CN=Users,YOUR_COMPANY_DC group_objectclass: group group_id_attribute: cn group_name_attribute: cn use_pool: True user_enabled_attribute: userAccountControl user_enabled_mask: 2 user_enabled_default: 512 use_tls: True tls_req_cert: demand # if you are configuring multiple LDAP domains, and LDAP server certificates are issued # by different authorities, make sure that you place certs for all the LDAP backend domains in the # cacert parameter as seen in this sample yml file so that all the certs are combined in a single CA file # and every LDAP domain configuration points to the combined CA file. # Note: # 1. Please be advised that every time a new ldap domain is configured, the single CA file gets overwritten # and hence ensure that you place certs for all the LDAP backend domains in the cacert parameter. # 2. There is a known issue on one cert per CA file per domain when the system processes # concurrent requests to multiple LDAP domains. Using the single CA file with all certs combined # shall get the system working properly.

tls_cacertfile: /etc/keystone/ssl/certs/all_ldapdomains_ca.pem

5. Add your new le to the local Git repository and commit the changes.

ardana > cd ~/openstackardana > git checkout siteardana > git add -Aardana > git commit -m "Adding LDAP server integration config"

6. Run the configuration processor and deployment preparation playbooks to validate theYAML les and prepare the environment for configuration.


ardana > cd ~/openstack/ardana/ansibleardana > ansible-playbook -i hosts/localhost config-processor-run.ymlardana > ansible-playbook -i hosts/localhost ready-deployment.yml

7. Run the keystone reconfiguration playbook to implement your changes, passing the newlycreated YAML le as an argument to the -e@FILE_PATH parameter:

ardana > cd ~/scratch/ansible/next/ardana/ansibleardana > ansible-playbook -i hosts/verb_hosts keystone-reconfigure.yml \ -e@/var/lib/ardana/openstack/my_cloud/config/keystone/keystone_configure_ldap_my.yml

To integrate your SUSE OpenStack Cloud cloud with multiple domains, repeat these stepsstarting from Step 3 for each domain.


3 Cloud Lifecycle Manager Admin UI User Guide

The Cloud Lifecycle Manager Admin UI is a web-based GUI for viewing and managing the con-figuration of an installed cloud. After successfully deploying the cloud with the Install UI, thefinal screen displays a link to the CLM Admin UI. (For example, see Book “Deployment Guide us-ing Cloud Lifecycle Manager”, Chapter 21 “Installing with the Install UI”, Section 21.5 “Running the Install

UI”, Cloud Deployment Successful). Usually the URL associated with this link is https://DEPLOY-ER_MGMT_NET_IP:9085 , although it may be different depending on the cloud configuration andthe installed version of SUSE OpenStack Cloud.

3.1 Accessing the Admin UIIn a browser, go to https://DEPLOYER_MGMT_NET_IP:9085 .

The DEPLOYER_MGMT_NET_IP:PORT_NUMBER is not necessarily the same for all installations, andcan be displayed with the following command:

ardana > openstack endpoint list --service ardana --interface admin -c URL

Accessing the Cloud Lifecycle Manager Admin UI requires access to the MANAGEMENT networkthat was configured when the Cloud was deployed. Access to this network is necessary to be ableto access the Cloud Lifecycle Manager Admin UI and log in. Depending on the network setup,it may be necessary to use an SSH tunnel similar to what is recommended in Book “DeploymentGuide using Cloud Lifecycle Manager”, Chapter 21 “Installing with the Install UI”, Section 21.5 “Running

the Install UI”. The Admin UI requires keystone and HAProxy to be running and to be accesible.If keystone or HAProxy are not running, cloud reconfiguration is limited to the command line.

Logging in requires a keystone user. If the user is not an admin on the default domain and oneor more projects, the Cloud Lifecycle Manager Admin UI will not display information about theCloud and may present errors.

36 Accessing the Admin UI SUSE OpenStack Cloud 9

FIGURE 3.1: CLOUD LIFECYCLE MANAGER ADMIN UI LOGIN PAGE

3.2 Admin UI Pages

3.2.1 Services

Services pages relay information about the various OpenStack and other services that have beendeployed as part of the cloud. Service information displays the list of services registered withkeystone and the endpoints associated with those services. The information is equivalent torunning the command openstack endpoint list .

The Service Information table contains the following information, based on how the serviceis registered with keystone:

Name

The name of the service, this may be an OpenStack code name

Description

Service description, for some services this is a repeat of the name

Endpoints

37 Admin UI Pages SUSE OpenStack Cloud 9

Services typically have 1 or more endpoints that are accessible to make API calls. The mostcommon configuration is for a service to have Admin , Public , and Internal endpoints,with each intended for access by consumers corresponding to the type of endpoint.

Region

Service endpoints are part of a region. In multi-region clouds, some services will haveendpoints in multiple regions.

FIGURE 3.2: CLOUD LIFECYCLE MANAGER ADMIN UI SERVICE INFORMATION

3.2.2 Packages

The Packages tab displays packages that are part of the SUSE OpenStack Cloud product.

The SUSE Cloud Packages table contains the following:

Name

The name of the SUSE Cloud package

Version

The version of the package which is installed in the Cloud

38 Packages SUSE OpenStack Cloud 9

FIGURE 3.3: CLOUD LIFECYCLE MANAGER ADMIN UI SUSE CLOUD PACKAGE

NotePackages with the venv- prefix denote the version of the specific OpenStack package thatis deployed. The release name can be determined from the OpenStack Releases (https://releases.openstack.org/) page.

3.2.3 Configuration

The Configuration tab displays services that are deployed in the cloud and the configuration lesassociated with those services. Services may be reconfigured by editing the .j2 les listed andclicking the Update button.

This page also provides the ability to set up SUSE Enterprise Storage Integration after initialdeployment.

39 Configuration SUSE OpenStack Cloud 9

https://releases.openstack.org/https://releases.openstack.org/

FIGURE 3.4: CLOUD LIFECYCLE MANAGER ADMIN UI SUSE SERVICE CONFIGURATION

Clicking one of the listed configuration les opens the le editor where changes can be made.Asterisks identify les that have been edited but have not had their updates applied to the cloud.

FIGURE 3.5: CLOUD LIFECYCLE MANAGER ADMIN UI SUSE SERVICE CONFIGURATION EDITOR


After editing the service configuration, click the Update button to begin deploying configurationchanges to the cloud. The status of those changes will be streamed to the UI.

Configure SUSE Enterprise Storage After Initial Deployment

A link to the settings.yml le is available under the ses selection on the Configuration tab.

To set up SUSE Enterprise Storage Integration:

1. Click on the link to edit the settings.yml le.

2. Uncomment the ses_config_path parameter, specify the location on the deployer hostcontaining the ses_config.yml le, and save the settings.yml le.

3. If the ses_config.yml le does not yet exist in that location on the deployer host, a newlink will appear for uploading a le from your local workstation.

4. When ses_config.yml is present on the deployer host, it will appear in the ses sectionof the Configuration tab and can be edited directly there.

NoteIf the cloud is configured using self-signed certificates, the streaming status updates (in-cluding the log) may be interupted and require a reload of the CLM Admin UI. See Book“Security Guide”, Chapter 8 “Transport Layer Security (TLS) Overview”, Section 8.2 “TLS Configura-

tion” for details on using signed certificates.


FIGURE 3.6: CLOUD LIFECYCL

Operations Guide CLM - SUSE Linux · 5.4 Identity Service Token Validation Example126 5.5...

Documents

Transcript of Operations Guide CLM - SUSE Linux · 5.4 Identity Service Token Validation Example126 5.5...