Post on 08-Sep-2014
description
© 2014 Nippon Telegraph and Telephone Corporation
AUTOSCALING CLOUD FOUNDRY WITH BOSH Yudai Iwasaki NTT Service Innovation Laboratory Group
CF Summit 2014
© 2014 Nippon Telegraph and Telephone Corporation
• Core developer of Cloudn PaaS • Working on Cloud Foundry since 2012
– Nise BOSH, BOSH CloudStack CPI and BOSH AutoScaler • Twitter: @i_yudai
© 2014 Nippon Telegraph and Telephone Corporation
Agenda • Problem – System load is not fixed
• Solution – BOSH AutoScaler
• Getting Started
© 2014 Nippon Telegraph and Telephone Corporation
Problem: System Load is
Not Fixed
© 2014 Nippon Telegraph and Telephone Corporation
Daily, Weekly, Monthly, and Yearly Peaks
© 2014 Nippon Telegraph and Telephone Corporation
Daily, Weekly, Monthly, and Yearly Peaks
© 2014 Nippon Telegraph and Telephone Corporation
Manually Scaling? Always monitor load?
© 2014 Nippon Telegraph and Telephone Corporation
Impossible No one wants to do such a boring job
© 2014 Nippon Telegraph and Telephone Corporation
Solution: BOSH AutoScaler
© 2014 Nippon Telegraph and Telephone Corporation
BOSH AutoScaler • Extension for BOSH
• Flexible scaling policies in deployment manifest files
• Special support for Cloud Foundry
© 2014 Nippon Telegraph and Telephone Corporation
Architecture Overview
© 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler (with CF Plugin)
Heartbeat
Heartbeat (subscribe)
process (BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy (PUT /deployments)
matching rules against log periodically Logging metrics（@buffers） run()
Agent Agent
Agent Agent
Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF
CF Component CF Component
CF Component
Collector
TSDB
Varz
Load policies (GET /deployments)
© 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler (with CF Plugin)
Heartbeat
Heartbeat (subscribe)
process (BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy (PUT /deployments)
matching rules against log periodically Logging metrics（@buffers） run()
Agent Agent
Agent Agent
Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF
CF Component CF Component
CF Component
Collector
TSDB
Varz
Load policies (GET /deployments)
© 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler (with CF Plugin)
Heartbeat
Heartbeat (subscribe)
process (BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy (PUT /deployments)
matching rules against log periodically Logging metrics（@buffers） run()
Agent Agent Agent Agent Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF CF Component
CF Component CF Component
Collector
TSDB
Varz
Load policies (GET /deployments)
© 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler (with CF Plugin)
Heartbeat
Heartbeat (subscribe)
process (BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy (PUT /deployments)
matching rules against log periodically Logging metrics（@buffers） run()
Agent Agent Agent Agent Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF CF Component
CF Component CF Component
Collector
TSDB
Varz
Load policies (GET /deployments)
© 2014 Nippon Telegraph and Telephone Corporation
Policy Definitions
© 2014 Nippon Telegraph and Telephone Corporation
Defining policies
• Adding “scale” section • Defining conditions for
each job • Various condition classes
to define flexible policies
-‐-‐-‐ name: cf .... scale: jobs: -‐ name: router cooldown: 300 out: limit: 10 unit: 2 conditions: -‐ class: CpuAverage larger_than: 80 duration: 300 -‐ class: MemoryAverage larger_than: 90 duration: 300 in: limit: 3 conditions: -‐ class: CpuAverage smaller_than: 10 duration: 300 -‐ class: MemoryAverage smaller_than: 20 duration: 300
© 2014 Nippon Telegraph and Telephone Corporation
Conditions: BOSH Heartbeat • CpuAverage
– Average CPU percentage for duration • MemoryAverage
– Average memory percentage for duration • LoadAverage1
– Latest Load Average in 1 minute • LoadAverage5
– Latest Load Average in 5 minutes • LoadAverage15
– Latest Load Average in 15 minutes
© 2014 Nippon Telegraph and Telephone Corporation
Conditions: CF Plugin • CFVarzAverage
– Average Varz value for duration
DEA available_memory_ratio available_disk_ratio
Router total_routes latency.1m
HM9000 NumberOfRunningInstances NumberOfCrashedInstances
etcd SendingRequestRate ReceivingRequestRate
Loggregator Server receivedMessageCount numberOfWebsocketSinks
Cloud Controller connection_count threadqueue.num_waiting
© 2014 Nippon Telegraph and Telephone Corporation
Web UI
© 2014 Nippon Telegraph and Telephone Corporation
© 2014 Nippon Telegraph and Telephone Corporation
© 2014 Nippon Telegraph and Telephone Corporation
© 2014 Nippon Telegraph and Telephone Corporation
Getting Started It’s easy to plug in
© 2014 Nippon Telegraph and Telephone Corporation
https://github.com/nttlabs/bosh-scaler
© 2014 Nippon Telegraph and Telephone Corporation
-‐-‐-‐ # BOSH NATS settings nats: &nats uri: mbus://192.168.50.4:21084 user: nats pass: nats # BOSH Director REST API settings rest: &rest endpoint_uri: https://192.168.50.4:25555 user: scaler password: scaler disable_verify_certification: true
collectors: -‐ class: BoshNatsCollector bosh_nats: *nats bosh_rest: *rest -‐ class: CfVarzTsdbCollector port: 4567 listeners: -‐ class: BoshScaler bosh_rest: *rest interval: 60 # seconds buffer_size: 1000 ui: enable: true port: 8888
Configuration
© 2014 Nippon Telegraph and Telephone Corporation
... properties: collector: use_tsdb: true deployment_name: cf opentsdb: # your AutoScaler address address: 192.168.15.139 port: 4567
Cloud Foundry Manifest
© 2014 Nippon Telegraph and Telephone Corporation
Optimize Running Costs with BOSH AutoScaler