XenServer ResourceXenServer Resource

Agent ManagerAgent Manager

API LayerAPI LayerEC2EC2 CloudStackCloudStack

Virt

ual M

achi

ne M

anag

erVi

rtua

l Mac

hine

Man

ager

KVM Resource

KVM Resource

vSphere ResourcevSphere Resource

SRX Resource

SRX Resource F5 ResourceF5 Resource NetScaler

ResourceNetScaler Resource

OVM Resource

OVM Resource

Access ControlAccess Control

Stor

age

Man

ager

Stor

age

Man

ager

Net

wor

k M

anag

erN

etw

ork

Man

ager

Cons

ole

Prox

y M

anag

erCo

nsol

e Pr

oxy

Man

ager

Snap

shot

Man

ager

Snap

shot

Man

ager

Tem

plat

e M

anag

erTe

mpl

ate

Man

ager

Asyn

c Jo

b M

anag

erAs

ync

Job

Man

ager

…

Management Server

Kernel- Drives long running VM

operations- Syncs between resources

managed and DB- Generates events

Resource Management

Cluster Management

JobManagement DB

UI Cloud Portal CLI

Other Clients

Job Queue

Deployment Planning

Network Configurations

Network Elements

Hypervisor Gurus

DatabaseAccess

Alert & EventManagement

Plug

in A

PI

Hypervisor Resources

Network Resources

Storage Resources

ImageResources

SnapshotResources

REST API

OAM&P API End User API EC2 API Pluggable Service API EngineOther APIs

Security Adapters

Account Management Connectors

ACL & Authentication- Accounts, Domains, and Projects- ACL, limits checking

Services API

Serv

ices

API

Console Proxy Management

Template Access

HA

Usage Calculations

Additional Services

Event BusMessage Bus

Kernel Module

• Understands how to orchestrate long running processes (i.e. VM starts, Snapshot copies, Template propagation)

• Well defined process steps• Calls Plugin API to execute functionalities that

it needs

Plugins

• Various ways to add more capability to CloudStack

• Implements clearly defined interfaces• All operations must be idempotent• All calls are at transaction boundaries• Compiles only against the Plugin API module

Anatomy of a Plugin

ServerResource- Optional. Required if

Plugin needs to be co-located with the resource

- Implements translation layer to talk to resource

- Communicates with server component via JSON

ServerResource- Optional. Required if

Plugin needs to be co-located with the resource

- Implements translation layer to talk to resource

- Communicates with server component via JSON

Rest API- Optional. Required only if needs to expose configuration API to admin.

Plug

in A

PI

Data Access Layer

Implmentation

Anatomy of a Plugin

• Can be two jars: server component to be deployed on management server and an optional ServerResource component to be deployed co-located with the resource

• Server component can implement multiple Plugin APIs to affect its feature

• Can expose its own API through Pluggable Service so administrators can configure the plugin

• As an example, OVS plugin actually implements both NetworkGuru and NetworkElement

Plugin Interfaces Available• NetworkGuru – Implements various network isolation technologies

and ip address technologies• NetworkElement – Facilitate network services on network elements

to support a VM (i.e. DNS, DHCP, LB, VPN, Port Forwarding, etc)• DeploymentPlanner – Different algorithms to place a VM and

volumes.• Investigator – Ways to find out if a host is down or VM is down.• Fencer – Ways to fence off a VM if the state is unknown• UserAuthenticator – Methods of authenticating a user• SecurityChecker – ACL access• HostAllocator – Provides different ways to allocate host• StoragePoolAllocator – Provides different ways to allocate volumes

Adding a Plugin to CloudStack

• Components are configured though components.xml

• Supports DAO, Manager, and Adapter patterns• Open to other component frameworks (OSGi a

possibility)

Components.xml Example<components.xml> <system-integrity-checker class="com.cloud.upgrade.DatabaseUpgradeChecker"> <checker name="ManagementServerNode" class="com.cloud.cluster.ManagementServerNode"/> <checker name="EncryptionSecretKeyChecker" class="com.cloud.utils.crypt.EncryptionSecretKeyChecker"/> <checker name="DatabaseIntegrityChecker" class="com.cloud.upgrade.DatabaseIntegrityChecker"/> <checker name="DatabaseUpgradeChecker" class="com.cloud.upgrade.PremiumDatabaseUpgradeChecker"/> </system-integrity-checker> <interceptor library="com.cloud.configuration.DefaultInterceptorLibrary"/> <management-server class="com.cloud.server.ManagementServerExtImpl" library="com.cloud.configuration.PremiumComponentLibrary"> <adapters key="com.cloud.storage.allocator.StoragePoolAllocator"> <adapter name="LocalStorage" class="com.cloud.storage.allocator.LocalStoragePoolAllocator"/> <adapter name="Storage" class="com.cloud.storage.allocator.FirstFitStoragePoolAllocator"/> </adapters> <pluggableservice name="VirtualRouterElementService" key="com.cloud.network.element.VirtualRouterElementService" class="com.cloud.network.element.VirtualRouterElement"/> </management-server></components.xml>

Kernel

Sequence Flow for deploy VMEnd User Rest API

SecurityCheckers

User VM Mgr

Network Mgr

Storage MgrJob

SchedulingVirtualMachine Mgr

Network Guru

Deploy VM

ACL Checks

Allocate Entity in CS

Allocate VM

Allocate NIC

Allocate Volume

Allocate IP

Schedules Deploy Job

Returns with job id, VM id

Query Job Result

Returns with job status

Sequence Flow for deploy VMJob Threads

Network Element

User VM Mgr

Network Mgr

Storage Mgr

VirtualMachine Mgr

Network Guru

Start VM

Start VM

Prepare Nics

Notify that Nic is about to be started in network

Reserve resources for Nic

Services APIServer

Resources

Start User VM

Agent Calls

Prepare Volumes

Template Mgr

Deployment

Planner

Get a Deployment Plan (Host and StoragePool)

Prepare template on Primary Storage

Agent Calls

Agent Start VM Call

Stores job result

ServerResource

• Translation layer between CloudStack commands and resource API

• May be Co-located with resource• Have no access to DB• API defined in JSON messages

DAO

• SQL generation done mostly in GenericDaoBase• Uses JPA annotations• Very little code to write for each individual DAO• Database Access Layer for Kernel• No support for more complicated features such as

fetch strategy• Welcome to use other types of ORM in other

modules but like to hear about preferred library. (Hibernate is out due to licensing issues)

Example DAO// ExampleVO.java@Entity@Table(name=“example”)public class ExampleVO { @Id @GeneratedValue(strategy= GenerationType.IDENTITY) @Column(name=“id”) long id;

@Column(name=“name”) String name;

@Column(name=“value”) String value;}

// ExampleDao.javapublic interface ExampleDao extends GenericDao<ExampleVO, Long> {}

// ExampleDaoImpl.java@Local(value=ExampleDao.class)public class ExampleDaoImpl extends GenericDaoBase<ExampleVO, Long> implements ExampleDao {

protected ExampleDaoImpl() { }}

Triggering High Availability

VM HA are triggered via the following methods:• VM Sync detects out of band VM death• Resource Management detects that a resource is

unreachable and its state can not be determined.• VM start/stop has been sent to the resource but

resource does not return• Details of how high availability is done is at

http://docs.cloudstack.org/CloudStack_Documentation/Design_Documents/CloudStack_High_Availability_-_Developer's_Guide

High Availability Future

• Moving toward using the native HA capability of the hypervisor.

• Looking to do more in the DRS area to coordinate recovery of wide spread outage.

VM Sync• Currently a sync of VM state, not entire VM• VM Sync happens between management server and hypervisor resources• Peer-to-peer sync• Hypervisor DB is considered to be the DB of truth• Two steps:

– Full Sync– Intermittent delta sync

• Establishes full sync when first connecting to the hypervisor resource• After full sync, hypervisor resource keeps track of the last sync results and only

report out of band changes on delta sync• Utilizes the most abundant resources in data center: CPU and memory• Conserve the most scarce resource: DB connections• Virtually no DB connections utilized during delta sync unless there are out of

band changes.

Storage

Zone-Level Layer 3 Switch

Pod 2 Pod N

…

Private Network

Computing Server 1

Computing Server 3

Computing Server 2

Computing Server 4

Pod-Level Layer-2 Switch

Primary Storage

Primary Storage

Pod 1

Scale-Out NFS

Primary Storage

Clus

ter 2

Clus

ter 1

Storage

• CloudStack supports two types of storage– Primary Storage: block device to the VM– WORM Storage: Secondary or Object Store for

templates, ISO, and snapshot archiving• Primary storage is high on IOPs (expensive)• Secondary storage is high on capacity (cheap)• CloudStack manages the storage between the

two to achieve maximum benefit and resiliency

Disk Offering

• Disk Offering is how disks are offered to the end user

• Disk Offering has storage tags which can be used to implementing storage tiering

• Service Offering actually contains a disk offering for the root disk

Snapshots

• Snapshots are used as backups• Taken on the primary storage and moved to

secondary storage• Full snapshots on VmWare and KVM. Need

help.• Incremental snapshots on XenServer

XenServer Snapshot

Base Copy

TemplateD1

S1D2

VM S2

Full1

S11

S12

S13

Full2

S21

S22

S23

Storage Future

• Secondary Storage multi-homed in Pod• Object Store for between zones• Direct Access to Storage for enterprise• Storage Plugins