Autoscaling Cloud Foundry with BOSH (Cloud Foundry Summit 2014)

© 2014 Nippon Telegraph and Telephone Corporation

AUTOSCALING CLOUD FOUNDRY WITH BOSH Yudai Iwasaki NTT Service Innovation Laboratory Group

CF Summit 2014


•  Core developer of Cloudn PaaS •  Working on Cloud Foundry since 2012

–  Nise BOSH, BOSH CloudStack CPI and BOSH AutoScaler •  Twitter: @i_yudai


Agenda •  Problem – System load is not fixed

•  Solution – BOSH AutoScaler

•  Getting Started


Problem: System Load is

Not Fixed


Daily, Weekly, Monthly, and Yearly Peaks


Manually Scaling? Always monitor load?


Impossible No one wants to do such a boring job


Solution: BOSH AutoScaler


BOSH AutoScaler •  Extension for BOSH

•  Flexible scaling policies in deployment manifest files

•  Special support for Cloud Foundry


Architecture Overview


Director

BOSH

BOSH Client

BOSH Scaler (Listener)

NATS

BOSH NATS Collector

AutoScaler (with CF Plugin)

Heartbeat

Heartbeat (subscribe)

process (BOSH::Monitor::Event::Heartbeat)

Event Processor

process (BOSH::Monitor::Event::Base)

Deploy (PUT /deployments)

matching rules against log periodically Logging metrics（@buffers） run()

Agent Agent

Agent Agent

Agent

CF Varz Collector

process (Scaler::CfVarzMetric)

CF

CF Component CF Component

CF Component

Collector

TSDB

Varz

Load policies (GET /deployments)


Director

BOSH

BOSH Client

BOSH Scaler (Listener)

NATS

BOSH NATS Collector

AutoScaler (with CF Plugin)

Heartbeat

Heartbeat (subscribe)

process (BOSH::Monitor::Event::Heartbeat)

Event Processor

process (BOSH::Monitor::Event::Base)

Deploy (PUT /deployments)

matching rules against log periodically Logging metrics（@buffers） run()

Agent Agent Agent Agent Agent

CF Varz Collector

process (Scaler::CfVarzMetric)

CF CF Component

CF Component CF Component

Collector

TSDB

Varz

Load policies (GET /deployments)


Policy Definitions


Defining policies

•  Adding “scale” section •  Defining conditions for

each job •  Various condition classes

to define flexible policies

-‐-‐-‐ name: cf .... scale: jobs: -‐ name: router cooldown: 300 out: limit: 10 unit: 2 conditions: -‐ class: CpuAverage larger_than: 80 duration: 300 -‐ class: MemoryAverage larger_than: 90 duration: 300 in: limit: 3 conditions: -‐ class: CpuAverage smaller_than: 10 duration: 300 -‐ class: MemoryAverage smaller_than: 20 duration: 300


Conditions: BOSH Heartbeat •  CpuAverage

–  Average CPU percentage for duration •  MemoryAverage

–  Average memory percentage for duration •  LoadAverage1

–  Latest Load Average in 1 minute •  LoadAverage5

–  Latest Load Average in 5 minutes •  LoadAverage15

–  Latest Load Average in 15 minutes


Conditions: CF Plugin •  CFVarzAverage

–  Average Varz value for duration

DEA available_memory_ratio available_disk_ratio

Router total_routes latency.1m

HM9000 NumberOfRunningInstances NumberOfCrashedInstances

etcd SendingRequestRate ReceivingRequestRate

Loggregator Server receivedMessageCount numberOfWebsocketSinks

Cloud Controller connection_count threadqueue.num_waiting


Web UI


Getting Started It’s easy to plug in


https://github.com/nttlabs/bosh-scaler


-‐-‐-‐ # BOSH NATS settings nats: &nats uri: mbus://192.168.50.4:21084 user: nats pass: nats # BOSH Director REST API settings rest: &rest endpoint_uri: https://192.168.50.4:25555 user: scaler password: scaler disable_verify_certification: true

collectors: -‐ class: BoshNatsCollector bosh_nats: *nats bosh_rest: *rest -‐ class: CfVarzTsdbCollector port: 4567 listeners: -‐ class: BoshScaler bosh_rest: *rest interval: 60 # seconds buffer_size: 1000 ui: enable: true port: 8888

Configuration


... properties: collector: use_tsdb: true deployment_name: cf opentsdb: # your AutoScaler address address: 192.168.15.139 port: 4567

Cloud Foundry Manifest


Optimize Running Costs with BOSH AutoScaler

Autoscaling Cloud Foundry with BOSH (Cloud Foundry Summit 2014)

Technology

Transcript of Autoscaling Cloud Foundry with BOSH (Cloud Foundry Summit 2014)