Matthias Runge

CentOS Opstools SIG - Update

February 1, 2019

1

CentOS Opstools SIG Update

● Intro● What is included● A closer look at performance monitoring● Taking the next step

Agenda

2

INTRODUCTION

3

CentOS Opstools SIG Update4

● Intended to support operators in second day tasks● a piece in the puzzle for RHOSP

Opstools - tools for operators

CentOS Opstools SIG Update5

Community and products

**

CentOS Opstools SIG Update6

Relationship with community and OSP repos

collectdOpenStack

CentOS opstools

CentOS MessagingRDO

QPID

Cloud SIG

Red Hat AMQ

oVirt

OSP OSP opstools

RHV

EPEL

CENTRALIZED LOGGING

7

CentOS Opstools SIG Update8

Centralized logging

fluentd fluentd

fluentd

fluentd

elasticsearch

AVAILABILITY MONITORING

9

CentOS Opstools SIG Update10

Availability monitoring - Sensu

PERFORMANCE MONITORING

11

CentOS Opstools SIG Update

● collectd● Graphite/CarbonDB● Grafana● Gnocchi

12

Performance monitoring (past)

collectd

collectd

collectd

graphite

GrafanaMonitoring node

CentOS Opstools SIG Update

● collectd 5.8.0, released Nov 17, 2017● release 5.8.1 in preparation● small resource usage● around since 2005● about 80 plugins, reporting mostly low level system information

○ bindings for Python, Java, exec, Perl● generates both metrics and events

13

collectd

CentOS Opstools SIG Update14

Performance monitoring (past) - HA

collectd

collectd

collectd

graphite

GrafanaMonitoring node

CentOS Opstools SIG Update

Kubernetes is a portable, extensible open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation.

15

Kubernetes

CentOS Opstools SIG Update16

Prometheus

CentOS Opstools SIG Update

● Decouple Prometheus instances from configuration

17

Prometheus operator

CentOS Opstools SIG Update

● QPID proton● dispatch router● qpid cpp broker

18

Apache QPID

qdr qdr qdrqdr

producer consumer

broker broker

CentOS Opstools SIG Update

infra node

19

Service assurance framework

collectd

qdr

qdr

qdr

qdr

Prometheus

grafana

telemetry consumer

kubernetes cluster

broker broker

CentOS Opstools SIG Update20

Grafana

CentOS Opstools SIG Update21

● Alarming rules defined in prometheus● alarm actions executed by alertmanager● - name: openstack-api slow

rules: - alert: openstack-api expr: avg_over_time(collectd_curl_response_time[30s]) > 0.2 for: 3m labels: severity: pager annotations: summary: openstack api slow

Alarming

OUTLOOK

22

CentOS Opstools SIG Update

● integration of “self healing” parts?● adding tools like vitrage, or doctor?● deprecating sensu in favor of alertmanager

23

Outlook

Questions?

mrunge@redhat.com

24

THANK YOU!

plus.google.com/+RedHat

linkedin.com/company/red-hat

youtube.com/user/RedHatVideos

facebook.com/redhatinc

twitter.com/RedHatNews

25