AutoScaling WordPress with Docker & AWS - WordPress Meetup Karlsruhe - Plesk
Exploring Magnum and Senlin integration for autoscaling containers
Transcript of Exploring Magnum and Senlin integration for autoscaling containers
Exploring Magnum and Senlin integration 1
Exploring Magnum and Senlin integration for autoscaling containers
Hongbin Lu (Huawei) Jay Lau, Ton Ngo (IBM)Qiming Teng, Julio Ruano (IBM)
Exploring Magnum and Senlin integration 2
ContributorsZilian Ji, Linjuan Xia, Shijia E, Peiyu Lin, Tianyuan Zhuang, Ji Jiang, Yongkui Wang
Exploring Magnum and Senlin integration 3
Talk Outline
1. Use cases2. Magnum3. Senlin4. How it works5. Demo
Exploring Magnum and Senlin integration 5
Use case
• Shared COE cluster– Autoscale on some policies (utilization, etc.) – Scale number of nodes
• Multiple apps on cluster– Autoscale on some policies (request rate, etc.) – Scale number of containers
• Example: – 2, 3-tiered web apps: handle requests– SLA determines policy
Exploring Magnum and Senlin integration 6
Scaling scenarios (1)
node
c c c
node
c cc
node
c scale app
scale cluster
Single app on cluster• scale out containers as needed• scale cluster by utilization• coordination = utilization
Exploring Magnum and Senlin integration 7
Scaling scenarios (2)
node
c
node
c c
node
c c c c scale 2 apps
Two apps on cluster• scale out containers as needed• scale cluster by utilization• coordination = utilization
scale clusterunlimitedscale clusterlimited
Exploring Magnum and Senlin integration 8
Scaling scenarios (3)
node
c c
node
scale 2 apps
scale clusterlimited
c
node
c c c c c c
Two apps on cluster, limited cluster size• blue scales first, retains containers• red scales later, fails• maybe acceptable
• what if red is critical and blue is not• inverted priority: blue < red
Exploring Magnum and Senlin integration 9
Scaling scenarios (4)
node
c c scale 2 apps
Limit on apps• Place max on blue • blue scales, red does not scale• cannot use full resources
scale clusterlimited
c
nodenode
c
limited
Exploring Magnum and Senlin integration 10
Scaling scenarios (5)
node
c c
node
scale 2 apps
scale clusterlimited
c
node
c c c c c cc c c c
Priority on apps• blue scales first, red scales later • blue scales back to free up resource for red • 2 ways linkage: container cluster host cluster
Exploring Magnum and Senlin integration 12
Instance
Magnum Overview
• Provision• Kubernetes• Docker Swarm• Mesos
• Scale• Add instances• Remove instances
• Manage• Pod• Service• Replication Controller• Container
Nova Instance
containercontainer
Magnum CLI
Magnum
Heat
Exploring Magnum and Senlin integration 14
Magnum: Autoscale Bay
1. Raw data: Magnum pulls raw
data from Bays.
2. Metrics: Metrics, like memory
utilization, are computed and sent
them to Ceilometer.
3. Alarm: Ceilometer triggers alarms
based on user-specified scenario
and metrics.
4. Scale: Heat scales the bays.
Ceilometer
Magnum Heat
Bay1) Raw data
2) Metrics 3) Alarm
4) Scale
Exploring Magnum and Senlin integration 15
Kubernetes: Autoscale Pod
1. Metrics: The autoscaler
periodically queries pods’ metrics.
2. Scale: Based on collected
metrics, the autoscaler uses a
built-in algorithm to trigger the
scaling.
3. Resize: Replication Controller
resize the pods.
HorizontalPod
Autoscaler
Pods
1) Metrics 3) Resize
Replication Controller
2) Scale
16
In discussion• Magnum AutoScaler
– A new component in Magnum• Scaling Bays• Scaling Applications
– Interacts with AutoScaler from COEs• Kubernetes AutoScaler as a driver for container-level scaling• Magnum responsible for scaling RCs without Kubernetes autoscaler
– Customizability• data sources• metrics and thresholds• standby bay nodes (i.e. resource pool)
Exploring Magnum and Senlin integration
Exploring Magnum and Senlin integration 17
Two-Levels of AutoScaling
Working
K8S Bay
NodeNode
K8S Master
Heapster
Kublet
cAdvisor
Kublet
cAdvisor
Storage Backend
RC AutoScaler
Pod
Magnum AutoScaler Service
K8S Driver
Collector
Analyzer
Operator
Senlin
Magnum Conductor
Standby
Swarm Driver
Mesos Driver
19
Clustering - A Missing Service in OpenStack
Exploring Magnum and Senlin integration
Scalable
Load-Balanced
Highly-Available
Manageable
......
of any (OpenStack) objects
Exploring Magnum and Senlin integration 20
Senlin Architecture
Senlin API
Senlin Client
REST RPC
Senlin EngineSenlin
Database
Policies
Profiles
Exploring Magnum and Senlin integration 21
Clustering/AutoScaling/LoadBalancing
Nova
Kubernetes
Heat
Ironic BareMetal
VMs
Stacks
Containers
Senlin
placement
deletion
scaling
health
load-balance
batching
Exploring Magnum and Senlin integration 22
Scaling A Cluster using Senlin
type: os.nova.serverversion: 1.0properties: image: fedora-21 flavor: 3 networks: - network: private userdata: ...
Profile
Cluster
type: senlin.policy.scalingversion: 1.0properties: event: CLUSTER_SCALE_IN adjustment: type: CHANGE_IN_CAPACITY number: 1 min_step: 1 best_effort: True
attach
Policy
type: senlin.policy.scalingversion: 1.0properties: event: CLUSTER_SCALE_OUT adjustment: . . .
attach
Policy
webhook
RESIZE
Exploring Magnum and Senlin integration 23
Policies
type: senlin.policy.scalingversion: 1.0properties: adjustment: min_step: 1 best_effort: true number: 1 type: CHANGE_IN_CAPACITY event: CLUSTER_SCALE_OUT
• Declarative• Can be mapped to from a standard• Example: SUR_Cluster(Add k8s Minion / Add Spark worker node)
Exploring Magnum and Senlin integration 24
Triggertype: CeilometerThresholdAlarmversion: 1.0rule: meter_name: memory_util comparison_operator: gt threshold: 70 period: 60 evaluation_periods: 1 statistic: avg query: - field: resource_metadata.cluster op: eq value: SUR_Cluster
• Generic abstraction• alarms in OpenStack
monitoring services• e.g. Ceilometer Alarm
Exploring Magnum and Senlin integration 26
Design goals
1. Coordinate scaling at apps and cluster level• Control flows in both direction• Apps -> Cluster: utilization, other methods• Cluster -> Apps: priority, other methods
2. Integrate scaling support from COE if available• Kubernetes autoscaler
3. Policy driven• Easy to express, manage complex cases
Exploring Magnum and Senlin integration
template
Outer Stack
Autoscaling: OpenStack-Only Solution (Today)
27
AutoScalingGroup Stack
Heat
Ceilometer
AH
PUp PDn
AL
ASGS1 S2 S3update
ASG
Exploring Magnum and Senlin integration
Senlin
profile
Autoscaling: OpenStack-Only Solution (New)
28
Cluster
Ceilometer
AHPUp
PDnAL
SCALE_IN
SCALE_OUT
Node1 Node2 Node3
webhookspolicies triggers
Exploring Magnum and Senlin integration
Autoscaling with Kubernetes
30
VMs
K8S Bay
NodeNode
Heapster
Kublet
cAdvisor
Kublet
cAdvisor
Pod
Senlin
Magnum
profile
CeilometerWorkload
policy