Monitoring Swift · Monitoring Swift OpenStack Summit, Austin 2016 Adam Takvam, Sr. Systems...

MonitoringSwiftOpenStackSummit, Austin2016

AdamTakvam,Sr.SystemsEngineerMartinLanner,EngagementManager

April28,2016

2 |SwiftStack Confidential

3

Overview

• Problems- Usage intelligence- Capacityplanning- Operational health- Audittrails

• Background- Methods: logs+systemmetrics- Interpretation ofmetrics- Actions:thresholds +alerting

• Swiftkeymonitoring concepts- Whattomonitor?- Howtomonitor

• Monitoring methods - demos- Logging:ELK- Trending/Forecasting:

Prometheus +Grafana- Systemmonitoring:Zabbix

|SwiftStack Confidential

4

It’sLinux!


5

PropertiesofSwift

• Distributed system

• Extremelydurable through replicationorErasure Coding

• Nosinglepointoffailure

• Evendistributionofdata

• Resilient

• Self-healing capabilities

• Cantakealotofabuseandnegligence

6

AnatomyofaMonitoringSolution

• Agent: Gathersmetricsonahostandeitherpushedoradvertisesthem- Logstash- PrometheusNodeExporter- ZabbixAgent- NagiosNRPE

• Aggregation Engines: Collects metrics fromagents andprovides an APIwith access toaggregated metric values- Nagios- Zabbix- Elasticsearch- Prometheus

• Visualizer: Renders graphs inahuman-friendlyformat for easy comprehension ofsystemstate- Kibana- Grafana

• Alerting: Uses metric thresholds totriggeralerts when metrics fall out ofan acceptablerange- AlertManager- PagerDuty


7

FormsofMonitoring

• Systemutilization: CPU,memory,diskI/O,network,auditingcycles,replicatortiming

• Performance:Transaction latency

• Errors:Invalidrequests orstates

• Outages:Servicefailures

• Featureusage:Understand CRUDoperations andtrafficpatterns

• Audittrail:Whodidwhatwhen?

MonitoringLifecycle

• Measurement

• Reporting

• Characterization

• Thresholds

• Alerting

• Rootcauseanalysis

• Remediation- Manual- Automated


Developing aMonitoring Strategy

8

Examplesofmonitoringmethods

• ELK: Usage intelligence- Who?- Agents- HTTPresponse codes- Errors- Audittrails

• Prometheus: Capacityplanning- Datagrowth- Trendinganalytics

• Zabbix: Operationalhealth- Network- CPU- RAM

9

KeyconceptsformonitoringSwift

• Cluster full- df- Datagrowth- Capacityplanning

• Networking- Availability- Saturation

• Proxystate- CPU- /healthcheck

• Auditingcycles

• Replicationcycletiming

10

LoadbalancerhealthchecksagainstSwiftproxyservers

demo@demo:~$ curl http://swift.swiftstack.oss/healthcheckOK


• Mostloadbalancers runICMPchecksagainstallIPsinitspoolbydefault

• Also,considerconfiguring theloadbalancer torunTCPchecksagainstSwift’s/healthcheck endpoint

Example:

11

AudittrailswithELK


12

Objectsizedistribution


13

DistributionofCRUDoperationsovertime


14

ZabbixtriggersforSwift


15

Zabbixnodememoryusage


16

Zabbixdriveutilizationevents


17

DiskI/O


18

ObjectReplicatorOperations


19

Prometheus+Grafanatrendingandforecasting


20

Alerting

ALERT StorageCritical24HoursIF sum(predict_linear(node_filesystem_free{

job='swiftstack',mountpoint=~"/srv/node/.*”}[1d]), 24*3600) < sum(node_filesystem_size{job="swiftstack",mountpoint=~"/srv/node/.*”}) * 0.2

FOR 1hLABELS {group="storage_admin“severity="critical“

}


Translation:Sendacriticalalerttoallmembersofthestorage_admin groupifthetotalavailablestoragecapacityisprojectedtobelessthan20%ofthetotalstoragecapacitywithinthenext24hoursandthatforecasthasheldtrueforatleast1hour,recalculatingevery5minutes(perserverconfig /notshown).

Example:

21

Q&A/Demo


22

Thankyou!


Monitoring Swift · Monitoring Swift OpenStack Summit, Austin 2016 Adam Takvam, Sr. Systems...

Documents

Transcript of Monitoring Swift · Monitoring Swift OpenStack Summit, Austin 2016 Adam Takvam, Sr. Systems...