VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models....
Transcript of VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models....
VMware vCloudreg Director trade
Resource Allocation ModelsT E C H N I C A L M A R K E T I N G D O C U M E N TAT I O NV 1 0 U P D AT E D A U G U S T 2 0 1 2
VMware vCloud Director Resource Allocation Models
T E C H N I C A L W H I T E P A P E R 2
Table of Contents
Purpose and Overview 4
Target Audience 4
Interpreting This Document 4
Reference Material 4
Resource Management Constructs 5
vCloud Director Constructs 5
vSphere Resource Constructs 6
vCloud Director Architecture 7
Architecting the Provider Virtual Datacenter 8
Organization Virtual Datacenters 9
Allocation Models 10
Pay-As-You-Go Model Settings 12
Expandable Reservation 14
Reservation Pool Model Settings 14
Expandable Reservation 16
Custom PerndashVirtual Machine Resource Settings 16
Allocation Pool Model Settings 17
Expandable Reservation 20
Review of Resource Distribution 20
Virtual Machine Distribution on Storage Level 21
Pay-As-You-Go Model 21
Reservation Pool Model 23
Allocation Pool Model 23
vCloud Allocation Models in Practice 28
Consumer Functional Requirements 28
Pay-As-You-Go Model 28
Reservation Pool Model 28
Allocation Pool Model 28
Catalog Organization Virtual Datacenter 29
vCloud System Virtual Datacenter 29
Test Cases 30
Single Allocation Model Tests 30
Single Allocation Model ndash Pay-As-You-Go 30
Single Allocation Model ndash Reservation Pool 33
Single Allocation Model ndash Allocation Pool 34
VMware vCloud Director Resource Allocation Models
T E C H N I C A L W H I T E P A P E R 3
Multiple Allocation Models 36
Default Resource Allocation Settings Test 37
Custom Resource Allocation Settings Test 38
About the Authors 41
T E C H N I C A L W H I T E P A P E R 4
VMware vCloud Director Resource Allocation Models
Purpose and OverviewVMware vCloudreg Directortrade enables users to build a private cloudndashbased infrastructure as a service (IaaS) offering within their organization By providing a secure on-demand ability for end users to deploy workloads companies can realize a level of agility previously thought impossible
Resource design for vCloud Director involves examining requirements to determine how best to partition and organize resources With the commoditization of infrastructure resources the ability to scale them becomes increasingly important When designing for vCloud Director keep in mind that the ultimate consumers of the product are the end users of the system Taking a top-down approach to vCloud Director design necessitates an understanding of the new abstractions introduced in the vCloud API and how they map to traditional VMware vSpherereg objects This paper was developed to provide additional insight and information as to how vCloud Director allocation models relate to vSphere resource management The paper reviews and discusses various design considerations and operational procedures
Target AudienceThe target audience of this document is individuals with a technical background who will be designing deploying or managing a vCloud Director infrastructure This includes but is not limited to technical consultants infrastructure architects IT managers implementation engineers partner engineers sales engineers and customer staff This solution brief is not intended to replace the VMware vCloud Architecture Toolkit It instead is meant to supplement knowledge and provide additional information As vCloud Director implementations have evolved there has been a noticeable increase in provider and consumer confusion about the vCloud Director allocation concepts as related to vSphere
Interpreting This DocumentThe overall structure of this case study is for the most part self-explanatory However throughout this document there are key points of particular importance that will be highlighted to the user These points will be identified with one of the following labels
bullNOTEndashGeneralpointofimportanceorafurtherexplanationwithinaparticularsection
bullEXAMPLEndashAspecificusecasetoillustrateaconceptwithinasection
The authors of this document assume that the reader is familiar with vCloud Director vSphere VMwarereg vCenter Servertrade VMware vSphere High Availability (vSphere HA) VMware vSphere Distributed Resource Schedulertrade (vSphere DRS) and VMware vSphere Storage DRStrade technology and terminology
Reference MaterialReading the following documents that have been used to develop this solution brief is recommended
bullvSphere5ResourceManagementGuide
bullVMwarevCloudArchitectureToolkitemsp
T E C H N I C A L W H I T E P A P E R 5
VMware vCloud Director Resource Allocation Models
Resource Management ConstructsvCloud Director introduces logical constructs to facilitate multitenancy and provide interoperability between vCloud Director instances built to the vCloud API standard
vCloud Director ConstructsTable 1 describes the logical constructs in vCloud Director that are used to abstract the underlying vSphere constructs
VCLOUD DIRECTOR CONSTRUCT
DESCRIPTION
Organization An organization is the unit of multitenancy that represents a single logical security boundary An organization contains users virtual datacenters and networks
Provider Virtual Datacenter A provider virtual datacenter (VDC) is a grouping of compute and storage resources from a single vCenter Server instance A provider VDC consists of a pool of physical compute resources and one or more datastores Multiple organizations can share provider VDC resources
Organization Virtual Datacenter An organization virtual datacenter is a subgrouping of compute and storage resources allocated from a provider VDC and assigned to a single organization An organization VDC is provisioned resources using vCloud Director resource allocation models These are represented in vSphere by ldquoresource poolsrdquo defined in Table 2
Resource Allocation Models Resource allocation models define how resources are provisioned to an organizationrsquos VDC from the provider VDC They also define how resources can be used when deploying virtual applications (vApps) within the organization VDC
vApp A vApp is a container for a distributed software solution and is the standard unit of deployment in vCloud Director A vCloud Director vApp is very different from a vSphere vApp in the manner it is instantiated and consumed in vCloud Director It enables power operations to be defined and specifically ordered It consists of one or more virtual machines and can be imported or exported as an OVF package A vCloud vApp can have additional vCloud-specific constructs such as vApp networks
Table 1 vCloud Director Constructs
T E C H N I C A L W H I T E P A P E R 6
VMware vCloud Director Resource Allocation Models
vSphere Resource Constructs Table 2 describes the resource management constructs in vSphere
VSPHERE RESOURCE CONSTRUCT
DESCRIPTION
Distributed Resource Scheduler Cluster
A vSphere DRS cluster is a collection of vSphere hosts with shared resources It must be created first to obtain the benefits of cluster-level resource management such as resource pools
Resource Pools A resource pool is a logical abstraction of physical resources for flexible management of resources Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources
Reservation A reservation specifies the guaranteed minimum physical resource allocation for a virtual machine or resource pool A power-on operation of a virtual machine (via vCenter Server or directly accessing the vSphere host with VMware vSphere Clienttrade) will succeed only if there are enough unreserved resources to satisfy the reservation needed by the virtual machine The server guarantees that amount even when the physical server is heavily loaded The reservation is expressed in physical units (megahertz or megabytes)
Limit Limit specifies an upper boundary for CPU or memory resources that can be allocated to a virtual machine or resource A server can allocate more than the reservation to a virtual machine but it never allocates more than the limit even if there are unused resources on the system The limit is expressed in concrete units (megahertz or megabytes) The default value for limits is ldquounlimitedrdquo When the memory value is unlimited the amount of memory configured for the virtual machine when it was created becomes its effective limit
Shares Shares specify the relative importance of a virtual machine or resource pool If a virtual machine has twice as many shares of a resource as another virtual machine it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources
Memory Overhead Reservation Memory overhead reservation is the amount of consumed memory being used for virtualization purposes Overhead memory includes space reserved for the virtual machine frame buffer and various virtualization data structures such as shadow page tables
Table 2 vSphere Resource Management Constructs
T E C H N I C A L W H I T E P A P E R 7
VMware vCloud Director Resource Allocation Models
vCloud Director ArchitectureThe vSphere infrastructure provides the foundation of the vCloud Director architecture Building a vCloud service requires an understanding of the individual constructs of the entire architecture
The vSphere layer provides a consumable set of resources available to the VMware vCloud environment and will present these resources to a provider VDC The provider VDC supplies an abstraction layer for resources As each organization VDC withdraws resources from its provider VDC the provider VDC creates the association between the organization VDC and resources at the vSphere layer The vCloud allocation model defined on each organization VDC dictates how the resources of the vSphere layer are consumed and it influences the availability of resources to the other organization VDCs sharing the same provider VDC
O
rgan
izat
ion
Virtual D
atacenter Allocation Model Provider Virtual Datacenter
vSphere R
esou
rces
ConstructInterdynamics
Figure 1 Cycle of vSphere and vCloud Construct Interdynamics
This interaction of vSphere and vCloud constructs determines the flexibility of the resource availability within the vCloud environment And eventually it is the resource availability that drives the consolidation ratio of the organization VDCs within a provider VDC The remaining chapters expand on the architecture of the provider VDC and how vCloud allocation models consume resources
T E C H N I C A L W H I T E P A P E R 8
VMware vCloud Director Resource Allocation Models
Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores
Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering
EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC
Cluster
Storage
Datastores
Provider Virtual Datacenter
Compute
ResourcePool
Figure 2 Provider VDC Resource Mapping
A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster
Provider VDC
Resource Pool
DRS Cluster
Map to
Figure 3 Mapping Provider VDC to Cluster or Resource Pool
By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool
The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources
T E C H N I C A L W H I T E P A P E R 9
VMware vCloud Director Resource Allocation Models
NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible
vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool
Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources
An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations
Organization VDC
Organization Customer 1
GoldProvider
BronzeProvider
Provider VDC
Organization VDC
Organization VDC
Provider VDC
Figure 4 Multiple Organization VDCs
Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models
bullPay-as-you-go
bullAllocation-pool
bullReservation-pool
T E C H N I C A L W H I T E P A P E R 1 0
VMware vCloud Director Resource Allocation Models
Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool
Figure 5 Overview vCloud Director Allocation Model
VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING
Allocation Limit
Guarantee Reservation
vCPU speed Limit on CPU virtual machine level
Table 3 Resource-Setting Taxonomy
The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both
T E C H N I C A L W H I T E P A P E R 1 1
VMware vCloud Director Resource Allocation Models
Resource Pool Resource Pool Resource Pool
VM VM VM VM VM VMVM
Pay-As-You-Go Model
Virtual MachinendashLevelSettings
Virtual Machinendash and Resource PoolndashLevel
Settings
Resource PoolndashLevelSettings
Reservation-Pool Model Allocation-Pool Model
VM VM
Figure 6 Overview of Allocation Models and vSphere Settings
Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo
NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC
Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC
The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Table 4 Provider VDC Resources
T E C H N I C A L W H I T E P A P E R 1 2
VMware vCloud Director Resource Allocation Models
Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults
VM VM VM
MemoryReservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
CPU Limit
CPU Reservation
Resource Pool
Figure 7 Pay-As-You-Go Model Default Settings
Figure 8 Pay-As-You-Go Model Default Settings User Interface
CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit
T E C H N I C A L W H I T E P A P E R 1 3
VMware vCloud Director Resource Allocation Models
EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent
Figure 9 Pay-As-You-Go Virtual Machine CPU Settings
Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface
EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface
Figure 10 Pay-As-You-Go Virtual Machine Memory Settings
T E C H N I C A L W H I T E P A P E R 1 4
VMware vCloud Director Resource Allocation Models
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool NA NA Unlimited NA Enabled
Virtual Machine 026GHz per vCPU
0 NA 100 NA
Table 5 Pay-As-You-Go Default Settings
A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent
Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
Figure 11 Reservation Pool Model Default Settings
T E C H N I C A L W H I T E P A P E R 1 5
VMware vCloud Director Resource Allocation Models
Figure 12 Reservation Pool Model Default Settings User Interface
NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)
The reservation pool model applies resource allocation settings on the resource pool only
CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level
Figure 13 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
VMware vCloud Director Resource Allocation Models
T E C H N I C A L W H I T E P A P E R 2
Table of Contents
Purpose and Overview 4
Target Audience 4
Interpreting This Document 4
Reference Material 4
Resource Management Constructs 5
vCloud Director Constructs 5
vSphere Resource Constructs 6
vCloud Director Architecture 7
Architecting the Provider Virtual Datacenter 8
Organization Virtual Datacenters 9
Allocation Models 10
Pay-As-You-Go Model Settings 12
Expandable Reservation 14
Reservation Pool Model Settings 14
Expandable Reservation 16
Custom PerndashVirtual Machine Resource Settings 16
Allocation Pool Model Settings 17
Expandable Reservation 20
Review of Resource Distribution 20
Virtual Machine Distribution on Storage Level 21
Pay-As-You-Go Model 21
Reservation Pool Model 23
Allocation Pool Model 23
vCloud Allocation Models in Practice 28
Consumer Functional Requirements 28
Pay-As-You-Go Model 28
Reservation Pool Model 28
Allocation Pool Model 28
Catalog Organization Virtual Datacenter 29
vCloud System Virtual Datacenter 29
Test Cases 30
Single Allocation Model Tests 30
Single Allocation Model ndash Pay-As-You-Go 30
Single Allocation Model ndash Reservation Pool 33
Single Allocation Model ndash Allocation Pool 34
VMware vCloud Director Resource Allocation Models
T E C H N I C A L W H I T E P A P E R 3
Multiple Allocation Models 36
Default Resource Allocation Settings Test 37
Custom Resource Allocation Settings Test 38
About the Authors 41
T E C H N I C A L W H I T E P A P E R 4
VMware vCloud Director Resource Allocation Models
Purpose and OverviewVMware vCloudreg Directortrade enables users to build a private cloudndashbased infrastructure as a service (IaaS) offering within their organization By providing a secure on-demand ability for end users to deploy workloads companies can realize a level of agility previously thought impossible
Resource design for vCloud Director involves examining requirements to determine how best to partition and organize resources With the commoditization of infrastructure resources the ability to scale them becomes increasingly important When designing for vCloud Director keep in mind that the ultimate consumers of the product are the end users of the system Taking a top-down approach to vCloud Director design necessitates an understanding of the new abstractions introduced in the vCloud API and how they map to traditional VMware vSpherereg objects This paper was developed to provide additional insight and information as to how vCloud Director allocation models relate to vSphere resource management The paper reviews and discusses various design considerations and operational procedures
Target AudienceThe target audience of this document is individuals with a technical background who will be designing deploying or managing a vCloud Director infrastructure This includes but is not limited to technical consultants infrastructure architects IT managers implementation engineers partner engineers sales engineers and customer staff This solution brief is not intended to replace the VMware vCloud Architecture Toolkit It instead is meant to supplement knowledge and provide additional information As vCloud Director implementations have evolved there has been a noticeable increase in provider and consumer confusion about the vCloud Director allocation concepts as related to vSphere
Interpreting This DocumentThe overall structure of this case study is for the most part self-explanatory However throughout this document there are key points of particular importance that will be highlighted to the user These points will be identified with one of the following labels
bullNOTEndashGeneralpointofimportanceorafurtherexplanationwithinaparticularsection
bullEXAMPLEndashAspecificusecasetoillustrateaconceptwithinasection
The authors of this document assume that the reader is familiar with vCloud Director vSphere VMwarereg vCenter Servertrade VMware vSphere High Availability (vSphere HA) VMware vSphere Distributed Resource Schedulertrade (vSphere DRS) and VMware vSphere Storage DRStrade technology and terminology
Reference MaterialReading the following documents that have been used to develop this solution brief is recommended
bullvSphere5ResourceManagementGuide
bullVMwarevCloudArchitectureToolkitemsp
T E C H N I C A L W H I T E P A P E R 5
VMware vCloud Director Resource Allocation Models
Resource Management ConstructsvCloud Director introduces logical constructs to facilitate multitenancy and provide interoperability between vCloud Director instances built to the vCloud API standard
vCloud Director ConstructsTable 1 describes the logical constructs in vCloud Director that are used to abstract the underlying vSphere constructs
VCLOUD DIRECTOR CONSTRUCT
DESCRIPTION
Organization An organization is the unit of multitenancy that represents a single logical security boundary An organization contains users virtual datacenters and networks
Provider Virtual Datacenter A provider virtual datacenter (VDC) is a grouping of compute and storage resources from a single vCenter Server instance A provider VDC consists of a pool of physical compute resources and one or more datastores Multiple organizations can share provider VDC resources
Organization Virtual Datacenter An organization virtual datacenter is a subgrouping of compute and storage resources allocated from a provider VDC and assigned to a single organization An organization VDC is provisioned resources using vCloud Director resource allocation models These are represented in vSphere by ldquoresource poolsrdquo defined in Table 2
Resource Allocation Models Resource allocation models define how resources are provisioned to an organizationrsquos VDC from the provider VDC They also define how resources can be used when deploying virtual applications (vApps) within the organization VDC
vApp A vApp is a container for a distributed software solution and is the standard unit of deployment in vCloud Director A vCloud Director vApp is very different from a vSphere vApp in the manner it is instantiated and consumed in vCloud Director It enables power operations to be defined and specifically ordered It consists of one or more virtual machines and can be imported or exported as an OVF package A vCloud vApp can have additional vCloud-specific constructs such as vApp networks
Table 1 vCloud Director Constructs
T E C H N I C A L W H I T E P A P E R 6
VMware vCloud Director Resource Allocation Models
vSphere Resource Constructs Table 2 describes the resource management constructs in vSphere
VSPHERE RESOURCE CONSTRUCT
DESCRIPTION
Distributed Resource Scheduler Cluster
A vSphere DRS cluster is a collection of vSphere hosts with shared resources It must be created first to obtain the benefits of cluster-level resource management such as resource pools
Resource Pools A resource pool is a logical abstraction of physical resources for flexible management of resources Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources
Reservation A reservation specifies the guaranteed minimum physical resource allocation for a virtual machine or resource pool A power-on operation of a virtual machine (via vCenter Server or directly accessing the vSphere host with VMware vSphere Clienttrade) will succeed only if there are enough unreserved resources to satisfy the reservation needed by the virtual machine The server guarantees that amount even when the physical server is heavily loaded The reservation is expressed in physical units (megahertz or megabytes)
Limit Limit specifies an upper boundary for CPU or memory resources that can be allocated to a virtual machine or resource A server can allocate more than the reservation to a virtual machine but it never allocates more than the limit even if there are unused resources on the system The limit is expressed in concrete units (megahertz or megabytes) The default value for limits is ldquounlimitedrdquo When the memory value is unlimited the amount of memory configured for the virtual machine when it was created becomes its effective limit
Shares Shares specify the relative importance of a virtual machine or resource pool If a virtual machine has twice as many shares of a resource as another virtual machine it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources
Memory Overhead Reservation Memory overhead reservation is the amount of consumed memory being used for virtualization purposes Overhead memory includes space reserved for the virtual machine frame buffer and various virtualization data structures such as shadow page tables
Table 2 vSphere Resource Management Constructs
T E C H N I C A L W H I T E P A P E R 7
VMware vCloud Director Resource Allocation Models
vCloud Director ArchitectureThe vSphere infrastructure provides the foundation of the vCloud Director architecture Building a vCloud service requires an understanding of the individual constructs of the entire architecture
The vSphere layer provides a consumable set of resources available to the VMware vCloud environment and will present these resources to a provider VDC The provider VDC supplies an abstraction layer for resources As each organization VDC withdraws resources from its provider VDC the provider VDC creates the association between the organization VDC and resources at the vSphere layer The vCloud allocation model defined on each organization VDC dictates how the resources of the vSphere layer are consumed and it influences the availability of resources to the other organization VDCs sharing the same provider VDC
O
rgan
izat
ion
Virtual D
atacenter Allocation Model Provider Virtual Datacenter
vSphere R
esou
rces
ConstructInterdynamics
Figure 1 Cycle of vSphere and vCloud Construct Interdynamics
This interaction of vSphere and vCloud constructs determines the flexibility of the resource availability within the vCloud environment And eventually it is the resource availability that drives the consolidation ratio of the organization VDCs within a provider VDC The remaining chapters expand on the architecture of the provider VDC and how vCloud allocation models consume resources
T E C H N I C A L W H I T E P A P E R 8
VMware vCloud Director Resource Allocation Models
Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores
Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering
EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC
Cluster
Storage
Datastores
Provider Virtual Datacenter
Compute
ResourcePool
Figure 2 Provider VDC Resource Mapping
A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster
Provider VDC
Resource Pool
DRS Cluster
Map to
Figure 3 Mapping Provider VDC to Cluster or Resource Pool
By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool
The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources
T E C H N I C A L W H I T E P A P E R 9
VMware vCloud Director Resource Allocation Models
NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible
vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool
Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources
An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations
Organization VDC
Organization Customer 1
GoldProvider
BronzeProvider
Provider VDC
Organization VDC
Organization VDC
Provider VDC
Figure 4 Multiple Organization VDCs
Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models
bullPay-as-you-go
bullAllocation-pool
bullReservation-pool
T E C H N I C A L W H I T E P A P E R 1 0
VMware vCloud Director Resource Allocation Models
Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool
Figure 5 Overview vCloud Director Allocation Model
VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING
Allocation Limit
Guarantee Reservation
vCPU speed Limit on CPU virtual machine level
Table 3 Resource-Setting Taxonomy
The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both
T E C H N I C A L W H I T E P A P E R 1 1
VMware vCloud Director Resource Allocation Models
Resource Pool Resource Pool Resource Pool
VM VM VM VM VM VMVM
Pay-As-You-Go Model
Virtual MachinendashLevelSettings
Virtual Machinendash and Resource PoolndashLevel
Settings
Resource PoolndashLevelSettings
Reservation-Pool Model Allocation-Pool Model
VM VM
Figure 6 Overview of Allocation Models and vSphere Settings
Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo
NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC
Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC
The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Table 4 Provider VDC Resources
T E C H N I C A L W H I T E P A P E R 1 2
VMware vCloud Director Resource Allocation Models
Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults
VM VM VM
MemoryReservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
CPU Limit
CPU Reservation
Resource Pool
Figure 7 Pay-As-You-Go Model Default Settings
Figure 8 Pay-As-You-Go Model Default Settings User Interface
CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit
T E C H N I C A L W H I T E P A P E R 1 3
VMware vCloud Director Resource Allocation Models
EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent
Figure 9 Pay-As-You-Go Virtual Machine CPU Settings
Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface
EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface
Figure 10 Pay-As-You-Go Virtual Machine Memory Settings
T E C H N I C A L W H I T E P A P E R 1 4
VMware vCloud Director Resource Allocation Models
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool NA NA Unlimited NA Enabled
Virtual Machine 026GHz per vCPU
0 NA 100 NA
Table 5 Pay-As-You-Go Default Settings
A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent
Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
Figure 11 Reservation Pool Model Default Settings
T E C H N I C A L W H I T E P A P E R 1 5
VMware vCloud Director Resource Allocation Models
Figure 12 Reservation Pool Model Default Settings User Interface
NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)
The reservation pool model applies resource allocation settings on the resource pool only
CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level
Figure 13 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
VMware vCloud Director Resource Allocation Models
T E C H N I C A L W H I T E P A P E R 3
Multiple Allocation Models 36
Default Resource Allocation Settings Test 37
Custom Resource Allocation Settings Test 38
About the Authors 41
T E C H N I C A L W H I T E P A P E R 4
VMware vCloud Director Resource Allocation Models
Purpose and OverviewVMware vCloudreg Directortrade enables users to build a private cloudndashbased infrastructure as a service (IaaS) offering within their organization By providing a secure on-demand ability for end users to deploy workloads companies can realize a level of agility previously thought impossible
Resource design for vCloud Director involves examining requirements to determine how best to partition and organize resources With the commoditization of infrastructure resources the ability to scale them becomes increasingly important When designing for vCloud Director keep in mind that the ultimate consumers of the product are the end users of the system Taking a top-down approach to vCloud Director design necessitates an understanding of the new abstractions introduced in the vCloud API and how they map to traditional VMware vSpherereg objects This paper was developed to provide additional insight and information as to how vCloud Director allocation models relate to vSphere resource management The paper reviews and discusses various design considerations and operational procedures
Target AudienceThe target audience of this document is individuals with a technical background who will be designing deploying or managing a vCloud Director infrastructure This includes but is not limited to technical consultants infrastructure architects IT managers implementation engineers partner engineers sales engineers and customer staff This solution brief is not intended to replace the VMware vCloud Architecture Toolkit It instead is meant to supplement knowledge and provide additional information As vCloud Director implementations have evolved there has been a noticeable increase in provider and consumer confusion about the vCloud Director allocation concepts as related to vSphere
Interpreting This DocumentThe overall structure of this case study is for the most part self-explanatory However throughout this document there are key points of particular importance that will be highlighted to the user These points will be identified with one of the following labels
bullNOTEndashGeneralpointofimportanceorafurtherexplanationwithinaparticularsection
bullEXAMPLEndashAspecificusecasetoillustrateaconceptwithinasection
The authors of this document assume that the reader is familiar with vCloud Director vSphere VMwarereg vCenter Servertrade VMware vSphere High Availability (vSphere HA) VMware vSphere Distributed Resource Schedulertrade (vSphere DRS) and VMware vSphere Storage DRStrade technology and terminology
Reference MaterialReading the following documents that have been used to develop this solution brief is recommended
bullvSphere5ResourceManagementGuide
bullVMwarevCloudArchitectureToolkitemsp
T E C H N I C A L W H I T E P A P E R 5
VMware vCloud Director Resource Allocation Models
Resource Management ConstructsvCloud Director introduces logical constructs to facilitate multitenancy and provide interoperability between vCloud Director instances built to the vCloud API standard
vCloud Director ConstructsTable 1 describes the logical constructs in vCloud Director that are used to abstract the underlying vSphere constructs
VCLOUD DIRECTOR CONSTRUCT
DESCRIPTION
Organization An organization is the unit of multitenancy that represents a single logical security boundary An organization contains users virtual datacenters and networks
Provider Virtual Datacenter A provider virtual datacenter (VDC) is a grouping of compute and storage resources from a single vCenter Server instance A provider VDC consists of a pool of physical compute resources and one or more datastores Multiple organizations can share provider VDC resources
Organization Virtual Datacenter An organization virtual datacenter is a subgrouping of compute and storage resources allocated from a provider VDC and assigned to a single organization An organization VDC is provisioned resources using vCloud Director resource allocation models These are represented in vSphere by ldquoresource poolsrdquo defined in Table 2
Resource Allocation Models Resource allocation models define how resources are provisioned to an organizationrsquos VDC from the provider VDC They also define how resources can be used when deploying virtual applications (vApps) within the organization VDC
vApp A vApp is a container for a distributed software solution and is the standard unit of deployment in vCloud Director A vCloud Director vApp is very different from a vSphere vApp in the manner it is instantiated and consumed in vCloud Director It enables power operations to be defined and specifically ordered It consists of one or more virtual machines and can be imported or exported as an OVF package A vCloud vApp can have additional vCloud-specific constructs such as vApp networks
Table 1 vCloud Director Constructs
T E C H N I C A L W H I T E P A P E R 6
VMware vCloud Director Resource Allocation Models
vSphere Resource Constructs Table 2 describes the resource management constructs in vSphere
VSPHERE RESOURCE CONSTRUCT
DESCRIPTION
Distributed Resource Scheduler Cluster
A vSphere DRS cluster is a collection of vSphere hosts with shared resources It must be created first to obtain the benefits of cluster-level resource management such as resource pools
Resource Pools A resource pool is a logical abstraction of physical resources for flexible management of resources Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources
Reservation A reservation specifies the guaranteed minimum physical resource allocation for a virtual machine or resource pool A power-on operation of a virtual machine (via vCenter Server or directly accessing the vSphere host with VMware vSphere Clienttrade) will succeed only if there are enough unreserved resources to satisfy the reservation needed by the virtual machine The server guarantees that amount even when the physical server is heavily loaded The reservation is expressed in physical units (megahertz or megabytes)
Limit Limit specifies an upper boundary for CPU or memory resources that can be allocated to a virtual machine or resource A server can allocate more than the reservation to a virtual machine but it never allocates more than the limit even if there are unused resources on the system The limit is expressed in concrete units (megahertz or megabytes) The default value for limits is ldquounlimitedrdquo When the memory value is unlimited the amount of memory configured for the virtual machine when it was created becomes its effective limit
Shares Shares specify the relative importance of a virtual machine or resource pool If a virtual machine has twice as many shares of a resource as another virtual machine it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources
Memory Overhead Reservation Memory overhead reservation is the amount of consumed memory being used for virtualization purposes Overhead memory includes space reserved for the virtual machine frame buffer and various virtualization data structures such as shadow page tables
Table 2 vSphere Resource Management Constructs
T E C H N I C A L W H I T E P A P E R 7
VMware vCloud Director Resource Allocation Models
vCloud Director ArchitectureThe vSphere infrastructure provides the foundation of the vCloud Director architecture Building a vCloud service requires an understanding of the individual constructs of the entire architecture
The vSphere layer provides a consumable set of resources available to the VMware vCloud environment and will present these resources to a provider VDC The provider VDC supplies an abstraction layer for resources As each organization VDC withdraws resources from its provider VDC the provider VDC creates the association between the organization VDC and resources at the vSphere layer The vCloud allocation model defined on each organization VDC dictates how the resources of the vSphere layer are consumed and it influences the availability of resources to the other organization VDCs sharing the same provider VDC
O
rgan
izat
ion
Virtual D
atacenter Allocation Model Provider Virtual Datacenter
vSphere R
esou
rces
ConstructInterdynamics
Figure 1 Cycle of vSphere and vCloud Construct Interdynamics
This interaction of vSphere and vCloud constructs determines the flexibility of the resource availability within the vCloud environment And eventually it is the resource availability that drives the consolidation ratio of the organization VDCs within a provider VDC The remaining chapters expand on the architecture of the provider VDC and how vCloud allocation models consume resources
T E C H N I C A L W H I T E P A P E R 8
VMware vCloud Director Resource Allocation Models
Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores
Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering
EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC
Cluster
Storage
Datastores
Provider Virtual Datacenter
Compute
ResourcePool
Figure 2 Provider VDC Resource Mapping
A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster
Provider VDC
Resource Pool
DRS Cluster
Map to
Figure 3 Mapping Provider VDC to Cluster or Resource Pool
By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool
The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources
T E C H N I C A L W H I T E P A P E R 9
VMware vCloud Director Resource Allocation Models
NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible
vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool
Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources
An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations
Organization VDC
Organization Customer 1
GoldProvider
BronzeProvider
Provider VDC
Organization VDC
Organization VDC
Provider VDC
Figure 4 Multiple Organization VDCs
Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models
bullPay-as-you-go
bullAllocation-pool
bullReservation-pool
T E C H N I C A L W H I T E P A P E R 1 0
VMware vCloud Director Resource Allocation Models
Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool
Figure 5 Overview vCloud Director Allocation Model
VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING
Allocation Limit
Guarantee Reservation
vCPU speed Limit on CPU virtual machine level
Table 3 Resource-Setting Taxonomy
The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both
T E C H N I C A L W H I T E P A P E R 1 1
VMware vCloud Director Resource Allocation Models
Resource Pool Resource Pool Resource Pool
VM VM VM VM VM VMVM
Pay-As-You-Go Model
Virtual MachinendashLevelSettings
Virtual Machinendash and Resource PoolndashLevel
Settings
Resource PoolndashLevelSettings
Reservation-Pool Model Allocation-Pool Model
VM VM
Figure 6 Overview of Allocation Models and vSphere Settings
Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo
NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC
Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC
The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Table 4 Provider VDC Resources
T E C H N I C A L W H I T E P A P E R 1 2
VMware vCloud Director Resource Allocation Models
Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults
VM VM VM
MemoryReservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
CPU Limit
CPU Reservation
Resource Pool
Figure 7 Pay-As-You-Go Model Default Settings
Figure 8 Pay-As-You-Go Model Default Settings User Interface
CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit
T E C H N I C A L W H I T E P A P E R 1 3
VMware vCloud Director Resource Allocation Models
EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent
Figure 9 Pay-As-You-Go Virtual Machine CPU Settings
Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface
EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface
Figure 10 Pay-As-You-Go Virtual Machine Memory Settings
T E C H N I C A L W H I T E P A P E R 1 4
VMware vCloud Director Resource Allocation Models
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool NA NA Unlimited NA Enabled
Virtual Machine 026GHz per vCPU
0 NA 100 NA
Table 5 Pay-As-You-Go Default Settings
A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent
Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
Figure 11 Reservation Pool Model Default Settings
T E C H N I C A L W H I T E P A P E R 1 5
VMware vCloud Director Resource Allocation Models
Figure 12 Reservation Pool Model Default Settings User Interface
NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)
The reservation pool model applies resource allocation settings on the resource pool only
CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level
Figure 13 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 4
VMware vCloud Director Resource Allocation Models
Purpose and OverviewVMware vCloudreg Directortrade enables users to build a private cloudndashbased infrastructure as a service (IaaS) offering within their organization By providing a secure on-demand ability for end users to deploy workloads companies can realize a level of agility previously thought impossible
Resource design for vCloud Director involves examining requirements to determine how best to partition and organize resources With the commoditization of infrastructure resources the ability to scale them becomes increasingly important When designing for vCloud Director keep in mind that the ultimate consumers of the product are the end users of the system Taking a top-down approach to vCloud Director design necessitates an understanding of the new abstractions introduced in the vCloud API and how they map to traditional VMware vSpherereg objects This paper was developed to provide additional insight and information as to how vCloud Director allocation models relate to vSphere resource management The paper reviews and discusses various design considerations and operational procedures
Target AudienceThe target audience of this document is individuals with a technical background who will be designing deploying or managing a vCloud Director infrastructure This includes but is not limited to technical consultants infrastructure architects IT managers implementation engineers partner engineers sales engineers and customer staff This solution brief is not intended to replace the VMware vCloud Architecture Toolkit It instead is meant to supplement knowledge and provide additional information As vCloud Director implementations have evolved there has been a noticeable increase in provider and consumer confusion about the vCloud Director allocation concepts as related to vSphere
Interpreting This DocumentThe overall structure of this case study is for the most part self-explanatory However throughout this document there are key points of particular importance that will be highlighted to the user These points will be identified with one of the following labels
bullNOTEndashGeneralpointofimportanceorafurtherexplanationwithinaparticularsection
bullEXAMPLEndashAspecificusecasetoillustrateaconceptwithinasection
The authors of this document assume that the reader is familiar with vCloud Director vSphere VMwarereg vCenter Servertrade VMware vSphere High Availability (vSphere HA) VMware vSphere Distributed Resource Schedulertrade (vSphere DRS) and VMware vSphere Storage DRStrade technology and terminology
Reference MaterialReading the following documents that have been used to develop this solution brief is recommended
bullvSphere5ResourceManagementGuide
bullVMwarevCloudArchitectureToolkitemsp
T E C H N I C A L W H I T E P A P E R 5
VMware vCloud Director Resource Allocation Models
Resource Management ConstructsvCloud Director introduces logical constructs to facilitate multitenancy and provide interoperability between vCloud Director instances built to the vCloud API standard
vCloud Director ConstructsTable 1 describes the logical constructs in vCloud Director that are used to abstract the underlying vSphere constructs
VCLOUD DIRECTOR CONSTRUCT
DESCRIPTION
Organization An organization is the unit of multitenancy that represents a single logical security boundary An organization contains users virtual datacenters and networks
Provider Virtual Datacenter A provider virtual datacenter (VDC) is a grouping of compute and storage resources from a single vCenter Server instance A provider VDC consists of a pool of physical compute resources and one or more datastores Multiple organizations can share provider VDC resources
Organization Virtual Datacenter An organization virtual datacenter is a subgrouping of compute and storage resources allocated from a provider VDC and assigned to a single organization An organization VDC is provisioned resources using vCloud Director resource allocation models These are represented in vSphere by ldquoresource poolsrdquo defined in Table 2
Resource Allocation Models Resource allocation models define how resources are provisioned to an organizationrsquos VDC from the provider VDC They also define how resources can be used when deploying virtual applications (vApps) within the organization VDC
vApp A vApp is a container for a distributed software solution and is the standard unit of deployment in vCloud Director A vCloud Director vApp is very different from a vSphere vApp in the manner it is instantiated and consumed in vCloud Director It enables power operations to be defined and specifically ordered It consists of one or more virtual machines and can be imported or exported as an OVF package A vCloud vApp can have additional vCloud-specific constructs such as vApp networks
Table 1 vCloud Director Constructs
T E C H N I C A L W H I T E P A P E R 6
VMware vCloud Director Resource Allocation Models
vSphere Resource Constructs Table 2 describes the resource management constructs in vSphere
VSPHERE RESOURCE CONSTRUCT
DESCRIPTION
Distributed Resource Scheduler Cluster
A vSphere DRS cluster is a collection of vSphere hosts with shared resources It must be created first to obtain the benefits of cluster-level resource management such as resource pools
Resource Pools A resource pool is a logical abstraction of physical resources for flexible management of resources Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources
Reservation A reservation specifies the guaranteed minimum physical resource allocation for a virtual machine or resource pool A power-on operation of a virtual machine (via vCenter Server or directly accessing the vSphere host with VMware vSphere Clienttrade) will succeed only if there are enough unreserved resources to satisfy the reservation needed by the virtual machine The server guarantees that amount even when the physical server is heavily loaded The reservation is expressed in physical units (megahertz or megabytes)
Limit Limit specifies an upper boundary for CPU or memory resources that can be allocated to a virtual machine or resource A server can allocate more than the reservation to a virtual machine but it never allocates more than the limit even if there are unused resources on the system The limit is expressed in concrete units (megahertz or megabytes) The default value for limits is ldquounlimitedrdquo When the memory value is unlimited the amount of memory configured for the virtual machine when it was created becomes its effective limit
Shares Shares specify the relative importance of a virtual machine or resource pool If a virtual machine has twice as many shares of a resource as another virtual machine it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources
Memory Overhead Reservation Memory overhead reservation is the amount of consumed memory being used for virtualization purposes Overhead memory includes space reserved for the virtual machine frame buffer and various virtualization data structures such as shadow page tables
Table 2 vSphere Resource Management Constructs
T E C H N I C A L W H I T E P A P E R 7
VMware vCloud Director Resource Allocation Models
vCloud Director ArchitectureThe vSphere infrastructure provides the foundation of the vCloud Director architecture Building a vCloud service requires an understanding of the individual constructs of the entire architecture
The vSphere layer provides a consumable set of resources available to the VMware vCloud environment and will present these resources to a provider VDC The provider VDC supplies an abstraction layer for resources As each organization VDC withdraws resources from its provider VDC the provider VDC creates the association between the organization VDC and resources at the vSphere layer The vCloud allocation model defined on each organization VDC dictates how the resources of the vSphere layer are consumed and it influences the availability of resources to the other organization VDCs sharing the same provider VDC
O
rgan
izat
ion
Virtual D
atacenter Allocation Model Provider Virtual Datacenter
vSphere R
esou
rces
ConstructInterdynamics
Figure 1 Cycle of vSphere and vCloud Construct Interdynamics
This interaction of vSphere and vCloud constructs determines the flexibility of the resource availability within the vCloud environment And eventually it is the resource availability that drives the consolidation ratio of the organization VDCs within a provider VDC The remaining chapters expand on the architecture of the provider VDC and how vCloud allocation models consume resources
T E C H N I C A L W H I T E P A P E R 8
VMware vCloud Director Resource Allocation Models
Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores
Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering
EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC
Cluster
Storage
Datastores
Provider Virtual Datacenter
Compute
ResourcePool
Figure 2 Provider VDC Resource Mapping
A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster
Provider VDC
Resource Pool
DRS Cluster
Map to
Figure 3 Mapping Provider VDC to Cluster or Resource Pool
By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool
The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources
T E C H N I C A L W H I T E P A P E R 9
VMware vCloud Director Resource Allocation Models
NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible
vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool
Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources
An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations
Organization VDC
Organization Customer 1
GoldProvider
BronzeProvider
Provider VDC
Organization VDC
Organization VDC
Provider VDC
Figure 4 Multiple Organization VDCs
Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models
bullPay-as-you-go
bullAllocation-pool
bullReservation-pool
T E C H N I C A L W H I T E P A P E R 1 0
VMware vCloud Director Resource Allocation Models
Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool
Figure 5 Overview vCloud Director Allocation Model
VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING
Allocation Limit
Guarantee Reservation
vCPU speed Limit on CPU virtual machine level
Table 3 Resource-Setting Taxonomy
The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both
T E C H N I C A L W H I T E P A P E R 1 1
VMware vCloud Director Resource Allocation Models
Resource Pool Resource Pool Resource Pool
VM VM VM VM VM VMVM
Pay-As-You-Go Model
Virtual MachinendashLevelSettings
Virtual Machinendash and Resource PoolndashLevel
Settings
Resource PoolndashLevelSettings
Reservation-Pool Model Allocation-Pool Model
VM VM
Figure 6 Overview of Allocation Models and vSphere Settings
Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo
NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC
Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC
The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Table 4 Provider VDC Resources
T E C H N I C A L W H I T E P A P E R 1 2
VMware vCloud Director Resource Allocation Models
Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults
VM VM VM
MemoryReservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
CPU Limit
CPU Reservation
Resource Pool
Figure 7 Pay-As-You-Go Model Default Settings
Figure 8 Pay-As-You-Go Model Default Settings User Interface
CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit
T E C H N I C A L W H I T E P A P E R 1 3
VMware vCloud Director Resource Allocation Models
EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent
Figure 9 Pay-As-You-Go Virtual Machine CPU Settings
Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface
EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface
Figure 10 Pay-As-You-Go Virtual Machine Memory Settings
T E C H N I C A L W H I T E P A P E R 1 4
VMware vCloud Director Resource Allocation Models
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool NA NA Unlimited NA Enabled
Virtual Machine 026GHz per vCPU
0 NA 100 NA
Table 5 Pay-As-You-Go Default Settings
A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent
Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
Figure 11 Reservation Pool Model Default Settings
T E C H N I C A L W H I T E P A P E R 1 5
VMware vCloud Director Resource Allocation Models
Figure 12 Reservation Pool Model Default Settings User Interface
NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)
The reservation pool model applies resource allocation settings on the resource pool only
CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level
Figure 13 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 5
VMware vCloud Director Resource Allocation Models
Resource Management ConstructsvCloud Director introduces logical constructs to facilitate multitenancy and provide interoperability between vCloud Director instances built to the vCloud API standard
vCloud Director ConstructsTable 1 describes the logical constructs in vCloud Director that are used to abstract the underlying vSphere constructs
VCLOUD DIRECTOR CONSTRUCT
DESCRIPTION
Organization An organization is the unit of multitenancy that represents a single logical security boundary An organization contains users virtual datacenters and networks
Provider Virtual Datacenter A provider virtual datacenter (VDC) is a grouping of compute and storage resources from a single vCenter Server instance A provider VDC consists of a pool of physical compute resources and one or more datastores Multiple organizations can share provider VDC resources
Organization Virtual Datacenter An organization virtual datacenter is a subgrouping of compute and storage resources allocated from a provider VDC and assigned to a single organization An organization VDC is provisioned resources using vCloud Director resource allocation models These are represented in vSphere by ldquoresource poolsrdquo defined in Table 2
Resource Allocation Models Resource allocation models define how resources are provisioned to an organizationrsquos VDC from the provider VDC They also define how resources can be used when deploying virtual applications (vApps) within the organization VDC
vApp A vApp is a container for a distributed software solution and is the standard unit of deployment in vCloud Director A vCloud Director vApp is very different from a vSphere vApp in the manner it is instantiated and consumed in vCloud Director It enables power operations to be defined and specifically ordered It consists of one or more virtual machines and can be imported or exported as an OVF package A vCloud vApp can have additional vCloud-specific constructs such as vApp networks
Table 1 vCloud Director Constructs
T E C H N I C A L W H I T E P A P E R 6
VMware vCloud Director Resource Allocation Models
vSphere Resource Constructs Table 2 describes the resource management constructs in vSphere
VSPHERE RESOURCE CONSTRUCT
DESCRIPTION
Distributed Resource Scheduler Cluster
A vSphere DRS cluster is a collection of vSphere hosts with shared resources It must be created first to obtain the benefits of cluster-level resource management such as resource pools
Resource Pools A resource pool is a logical abstraction of physical resources for flexible management of resources Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources
Reservation A reservation specifies the guaranteed minimum physical resource allocation for a virtual machine or resource pool A power-on operation of a virtual machine (via vCenter Server or directly accessing the vSphere host with VMware vSphere Clienttrade) will succeed only if there are enough unreserved resources to satisfy the reservation needed by the virtual machine The server guarantees that amount even when the physical server is heavily loaded The reservation is expressed in physical units (megahertz or megabytes)
Limit Limit specifies an upper boundary for CPU or memory resources that can be allocated to a virtual machine or resource A server can allocate more than the reservation to a virtual machine but it never allocates more than the limit even if there are unused resources on the system The limit is expressed in concrete units (megahertz or megabytes) The default value for limits is ldquounlimitedrdquo When the memory value is unlimited the amount of memory configured for the virtual machine when it was created becomes its effective limit
Shares Shares specify the relative importance of a virtual machine or resource pool If a virtual machine has twice as many shares of a resource as another virtual machine it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources
Memory Overhead Reservation Memory overhead reservation is the amount of consumed memory being used for virtualization purposes Overhead memory includes space reserved for the virtual machine frame buffer and various virtualization data structures such as shadow page tables
Table 2 vSphere Resource Management Constructs
T E C H N I C A L W H I T E P A P E R 7
VMware vCloud Director Resource Allocation Models
vCloud Director ArchitectureThe vSphere infrastructure provides the foundation of the vCloud Director architecture Building a vCloud service requires an understanding of the individual constructs of the entire architecture
The vSphere layer provides a consumable set of resources available to the VMware vCloud environment and will present these resources to a provider VDC The provider VDC supplies an abstraction layer for resources As each organization VDC withdraws resources from its provider VDC the provider VDC creates the association between the organization VDC and resources at the vSphere layer The vCloud allocation model defined on each organization VDC dictates how the resources of the vSphere layer are consumed and it influences the availability of resources to the other organization VDCs sharing the same provider VDC
O
rgan
izat
ion
Virtual D
atacenter Allocation Model Provider Virtual Datacenter
vSphere R
esou
rces
ConstructInterdynamics
Figure 1 Cycle of vSphere and vCloud Construct Interdynamics
This interaction of vSphere and vCloud constructs determines the flexibility of the resource availability within the vCloud environment And eventually it is the resource availability that drives the consolidation ratio of the organization VDCs within a provider VDC The remaining chapters expand on the architecture of the provider VDC and how vCloud allocation models consume resources
T E C H N I C A L W H I T E P A P E R 8
VMware vCloud Director Resource Allocation Models
Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores
Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering
EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC
Cluster
Storage
Datastores
Provider Virtual Datacenter
Compute
ResourcePool
Figure 2 Provider VDC Resource Mapping
A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster
Provider VDC
Resource Pool
DRS Cluster
Map to
Figure 3 Mapping Provider VDC to Cluster or Resource Pool
By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool
The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources
T E C H N I C A L W H I T E P A P E R 9
VMware vCloud Director Resource Allocation Models
NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible
vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool
Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources
An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations
Organization VDC
Organization Customer 1
GoldProvider
BronzeProvider
Provider VDC
Organization VDC
Organization VDC
Provider VDC
Figure 4 Multiple Organization VDCs
Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models
bullPay-as-you-go
bullAllocation-pool
bullReservation-pool
T E C H N I C A L W H I T E P A P E R 1 0
VMware vCloud Director Resource Allocation Models
Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool
Figure 5 Overview vCloud Director Allocation Model
VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING
Allocation Limit
Guarantee Reservation
vCPU speed Limit on CPU virtual machine level
Table 3 Resource-Setting Taxonomy
The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both
T E C H N I C A L W H I T E P A P E R 1 1
VMware vCloud Director Resource Allocation Models
Resource Pool Resource Pool Resource Pool
VM VM VM VM VM VMVM
Pay-As-You-Go Model
Virtual MachinendashLevelSettings
Virtual Machinendash and Resource PoolndashLevel
Settings
Resource PoolndashLevelSettings
Reservation-Pool Model Allocation-Pool Model
VM VM
Figure 6 Overview of Allocation Models and vSphere Settings
Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo
NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC
Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC
The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Table 4 Provider VDC Resources
T E C H N I C A L W H I T E P A P E R 1 2
VMware vCloud Director Resource Allocation Models
Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults
VM VM VM
MemoryReservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
CPU Limit
CPU Reservation
Resource Pool
Figure 7 Pay-As-You-Go Model Default Settings
Figure 8 Pay-As-You-Go Model Default Settings User Interface
CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit
T E C H N I C A L W H I T E P A P E R 1 3
VMware vCloud Director Resource Allocation Models
EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent
Figure 9 Pay-As-You-Go Virtual Machine CPU Settings
Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface
EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface
Figure 10 Pay-As-You-Go Virtual Machine Memory Settings
T E C H N I C A L W H I T E P A P E R 1 4
VMware vCloud Director Resource Allocation Models
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool NA NA Unlimited NA Enabled
Virtual Machine 026GHz per vCPU
0 NA 100 NA
Table 5 Pay-As-You-Go Default Settings
A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent
Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
Figure 11 Reservation Pool Model Default Settings
T E C H N I C A L W H I T E P A P E R 1 5
VMware vCloud Director Resource Allocation Models
Figure 12 Reservation Pool Model Default Settings User Interface
NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)
The reservation pool model applies resource allocation settings on the resource pool only
CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level
Figure 13 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 6
VMware vCloud Director Resource Allocation Models
vSphere Resource Constructs Table 2 describes the resource management constructs in vSphere
VSPHERE RESOURCE CONSTRUCT
DESCRIPTION
Distributed Resource Scheduler Cluster
A vSphere DRS cluster is a collection of vSphere hosts with shared resources It must be created first to obtain the benefits of cluster-level resource management such as resource pools
Resource Pools A resource pool is a logical abstraction of physical resources for flexible management of resources Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources
Reservation A reservation specifies the guaranteed minimum physical resource allocation for a virtual machine or resource pool A power-on operation of a virtual machine (via vCenter Server or directly accessing the vSphere host with VMware vSphere Clienttrade) will succeed only if there are enough unreserved resources to satisfy the reservation needed by the virtual machine The server guarantees that amount even when the physical server is heavily loaded The reservation is expressed in physical units (megahertz or megabytes)
Limit Limit specifies an upper boundary for CPU or memory resources that can be allocated to a virtual machine or resource A server can allocate more than the reservation to a virtual machine but it never allocates more than the limit even if there are unused resources on the system The limit is expressed in concrete units (megahertz or megabytes) The default value for limits is ldquounlimitedrdquo When the memory value is unlimited the amount of memory configured for the virtual machine when it was created becomes its effective limit
Shares Shares specify the relative importance of a virtual machine or resource pool If a virtual machine has twice as many shares of a resource as another virtual machine it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources
Memory Overhead Reservation Memory overhead reservation is the amount of consumed memory being used for virtualization purposes Overhead memory includes space reserved for the virtual machine frame buffer and various virtualization data structures such as shadow page tables
Table 2 vSphere Resource Management Constructs
T E C H N I C A L W H I T E P A P E R 7
VMware vCloud Director Resource Allocation Models
vCloud Director ArchitectureThe vSphere infrastructure provides the foundation of the vCloud Director architecture Building a vCloud service requires an understanding of the individual constructs of the entire architecture
The vSphere layer provides a consumable set of resources available to the VMware vCloud environment and will present these resources to a provider VDC The provider VDC supplies an abstraction layer for resources As each organization VDC withdraws resources from its provider VDC the provider VDC creates the association between the organization VDC and resources at the vSphere layer The vCloud allocation model defined on each organization VDC dictates how the resources of the vSphere layer are consumed and it influences the availability of resources to the other organization VDCs sharing the same provider VDC
O
rgan
izat
ion
Virtual D
atacenter Allocation Model Provider Virtual Datacenter
vSphere R
esou
rces
ConstructInterdynamics
Figure 1 Cycle of vSphere and vCloud Construct Interdynamics
This interaction of vSphere and vCloud constructs determines the flexibility of the resource availability within the vCloud environment And eventually it is the resource availability that drives the consolidation ratio of the organization VDCs within a provider VDC The remaining chapters expand on the architecture of the provider VDC and how vCloud allocation models consume resources
T E C H N I C A L W H I T E P A P E R 8
VMware vCloud Director Resource Allocation Models
Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores
Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering
EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC
Cluster
Storage
Datastores
Provider Virtual Datacenter
Compute
ResourcePool
Figure 2 Provider VDC Resource Mapping
A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster
Provider VDC
Resource Pool
DRS Cluster
Map to
Figure 3 Mapping Provider VDC to Cluster or Resource Pool
By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool
The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources
T E C H N I C A L W H I T E P A P E R 9
VMware vCloud Director Resource Allocation Models
NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible
vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool
Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources
An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations
Organization VDC
Organization Customer 1
GoldProvider
BronzeProvider
Provider VDC
Organization VDC
Organization VDC
Provider VDC
Figure 4 Multiple Organization VDCs
Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models
bullPay-as-you-go
bullAllocation-pool
bullReservation-pool
T E C H N I C A L W H I T E P A P E R 1 0
VMware vCloud Director Resource Allocation Models
Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool
Figure 5 Overview vCloud Director Allocation Model
VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING
Allocation Limit
Guarantee Reservation
vCPU speed Limit on CPU virtual machine level
Table 3 Resource-Setting Taxonomy
The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both
T E C H N I C A L W H I T E P A P E R 1 1
VMware vCloud Director Resource Allocation Models
Resource Pool Resource Pool Resource Pool
VM VM VM VM VM VMVM
Pay-As-You-Go Model
Virtual MachinendashLevelSettings
Virtual Machinendash and Resource PoolndashLevel
Settings
Resource PoolndashLevelSettings
Reservation-Pool Model Allocation-Pool Model
VM VM
Figure 6 Overview of Allocation Models and vSphere Settings
Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo
NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC
Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC
The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Table 4 Provider VDC Resources
T E C H N I C A L W H I T E P A P E R 1 2
VMware vCloud Director Resource Allocation Models
Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults
VM VM VM
MemoryReservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
CPU Limit
CPU Reservation
Resource Pool
Figure 7 Pay-As-You-Go Model Default Settings
Figure 8 Pay-As-You-Go Model Default Settings User Interface
CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit
T E C H N I C A L W H I T E P A P E R 1 3
VMware vCloud Director Resource Allocation Models
EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent
Figure 9 Pay-As-You-Go Virtual Machine CPU Settings
Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface
EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface
Figure 10 Pay-As-You-Go Virtual Machine Memory Settings
T E C H N I C A L W H I T E P A P E R 1 4
VMware vCloud Director Resource Allocation Models
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool NA NA Unlimited NA Enabled
Virtual Machine 026GHz per vCPU
0 NA 100 NA
Table 5 Pay-As-You-Go Default Settings
A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent
Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
Figure 11 Reservation Pool Model Default Settings
T E C H N I C A L W H I T E P A P E R 1 5
VMware vCloud Director Resource Allocation Models
Figure 12 Reservation Pool Model Default Settings User Interface
NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)
The reservation pool model applies resource allocation settings on the resource pool only
CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level
Figure 13 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 7
VMware vCloud Director Resource Allocation Models
vCloud Director ArchitectureThe vSphere infrastructure provides the foundation of the vCloud Director architecture Building a vCloud service requires an understanding of the individual constructs of the entire architecture
The vSphere layer provides a consumable set of resources available to the VMware vCloud environment and will present these resources to a provider VDC The provider VDC supplies an abstraction layer for resources As each organization VDC withdraws resources from its provider VDC the provider VDC creates the association between the organization VDC and resources at the vSphere layer The vCloud allocation model defined on each organization VDC dictates how the resources of the vSphere layer are consumed and it influences the availability of resources to the other organization VDCs sharing the same provider VDC
O
rgan
izat
ion
Virtual D
atacenter Allocation Model Provider Virtual Datacenter
vSphere R
esou
rces
ConstructInterdynamics
Figure 1 Cycle of vSphere and vCloud Construct Interdynamics
This interaction of vSphere and vCloud constructs determines the flexibility of the resource availability within the vCloud environment And eventually it is the resource availability that drives the consolidation ratio of the organization VDCs within a provider VDC The remaining chapters expand on the architecture of the provider VDC and how vCloud allocation models consume resources
T E C H N I C A L W H I T E P A P E R 8
VMware vCloud Director Resource Allocation Models
Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores
Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering
EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC
Cluster
Storage
Datastores
Provider Virtual Datacenter
Compute
ResourcePool
Figure 2 Provider VDC Resource Mapping
A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster
Provider VDC
Resource Pool
DRS Cluster
Map to
Figure 3 Mapping Provider VDC to Cluster or Resource Pool
By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool
The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources
T E C H N I C A L W H I T E P A P E R 9
VMware vCloud Director Resource Allocation Models
NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible
vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool
Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources
An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations
Organization VDC
Organization Customer 1
GoldProvider
BronzeProvider
Provider VDC
Organization VDC
Organization VDC
Provider VDC
Figure 4 Multiple Organization VDCs
Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models
bullPay-as-you-go
bullAllocation-pool
bullReservation-pool
T E C H N I C A L W H I T E P A P E R 1 0
VMware vCloud Director Resource Allocation Models
Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool
Figure 5 Overview vCloud Director Allocation Model
VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING
Allocation Limit
Guarantee Reservation
vCPU speed Limit on CPU virtual machine level
Table 3 Resource-Setting Taxonomy
The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both
T E C H N I C A L W H I T E P A P E R 1 1
VMware vCloud Director Resource Allocation Models
Resource Pool Resource Pool Resource Pool
VM VM VM VM VM VMVM
Pay-As-You-Go Model
Virtual MachinendashLevelSettings
Virtual Machinendash and Resource PoolndashLevel
Settings
Resource PoolndashLevelSettings
Reservation-Pool Model Allocation-Pool Model
VM VM
Figure 6 Overview of Allocation Models and vSphere Settings
Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo
NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC
Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC
The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Table 4 Provider VDC Resources
T E C H N I C A L W H I T E P A P E R 1 2
VMware vCloud Director Resource Allocation Models
Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults
VM VM VM
MemoryReservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
CPU Limit
CPU Reservation
Resource Pool
Figure 7 Pay-As-You-Go Model Default Settings
Figure 8 Pay-As-You-Go Model Default Settings User Interface
CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit
T E C H N I C A L W H I T E P A P E R 1 3
VMware vCloud Director Resource Allocation Models
EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent
Figure 9 Pay-As-You-Go Virtual Machine CPU Settings
Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface
EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface
Figure 10 Pay-As-You-Go Virtual Machine Memory Settings
T E C H N I C A L W H I T E P A P E R 1 4
VMware vCloud Director Resource Allocation Models
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool NA NA Unlimited NA Enabled
Virtual Machine 026GHz per vCPU
0 NA 100 NA
Table 5 Pay-As-You-Go Default Settings
A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent
Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
Figure 11 Reservation Pool Model Default Settings
T E C H N I C A L W H I T E P A P E R 1 5
VMware vCloud Director Resource Allocation Models
Figure 12 Reservation Pool Model Default Settings User Interface
NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)
The reservation pool model applies resource allocation settings on the resource pool only
CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level
Figure 13 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 8
VMware vCloud Director Resource Allocation Models
Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores
Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering
EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC
Cluster
Storage
Datastores
Provider Virtual Datacenter
Compute
ResourcePool
Figure 2 Provider VDC Resource Mapping
A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster
Provider VDC
Resource Pool
DRS Cluster
Map to
Figure 3 Mapping Provider VDC to Cluster or Resource Pool
By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool
The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources
T E C H N I C A L W H I T E P A P E R 9
VMware vCloud Director Resource Allocation Models
NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible
vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool
Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources
An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations
Organization VDC
Organization Customer 1
GoldProvider
BronzeProvider
Provider VDC
Organization VDC
Organization VDC
Provider VDC
Figure 4 Multiple Organization VDCs
Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models
bullPay-as-you-go
bullAllocation-pool
bullReservation-pool
T E C H N I C A L W H I T E P A P E R 1 0
VMware vCloud Director Resource Allocation Models
Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool
Figure 5 Overview vCloud Director Allocation Model
VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING
Allocation Limit
Guarantee Reservation
vCPU speed Limit on CPU virtual machine level
Table 3 Resource-Setting Taxonomy
The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both
T E C H N I C A L W H I T E P A P E R 1 1
VMware vCloud Director Resource Allocation Models
Resource Pool Resource Pool Resource Pool
VM VM VM VM VM VMVM
Pay-As-You-Go Model
Virtual MachinendashLevelSettings
Virtual Machinendash and Resource PoolndashLevel
Settings
Resource PoolndashLevelSettings
Reservation-Pool Model Allocation-Pool Model
VM VM
Figure 6 Overview of Allocation Models and vSphere Settings
Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo
NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC
Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC
The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Table 4 Provider VDC Resources
T E C H N I C A L W H I T E P A P E R 1 2
VMware vCloud Director Resource Allocation Models
Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults
VM VM VM
MemoryReservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
CPU Limit
CPU Reservation
Resource Pool
Figure 7 Pay-As-You-Go Model Default Settings
Figure 8 Pay-As-You-Go Model Default Settings User Interface
CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit
T E C H N I C A L W H I T E P A P E R 1 3
VMware vCloud Director Resource Allocation Models
EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent
Figure 9 Pay-As-You-Go Virtual Machine CPU Settings
Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface
EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface
Figure 10 Pay-As-You-Go Virtual Machine Memory Settings
T E C H N I C A L W H I T E P A P E R 1 4
VMware vCloud Director Resource Allocation Models
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool NA NA Unlimited NA Enabled
Virtual Machine 026GHz per vCPU
0 NA 100 NA
Table 5 Pay-As-You-Go Default Settings
A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent
Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
Figure 11 Reservation Pool Model Default Settings
T E C H N I C A L W H I T E P A P E R 1 5
VMware vCloud Director Resource Allocation Models
Figure 12 Reservation Pool Model Default Settings User Interface
NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)
The reservation pool model applies resource allocation settings on the resource pool only
CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level
Figure 13 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 9
VMware vCloud Director Resource Allocation Models
NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible
vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool
Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources
An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations
Organization VDC
Organization Customer 1
GoldProvider
BronzeProvider
Provider VDC
Organization VDC
Organization VDC
Provider VDC
Figure 4 Multiple Organization VDCs
Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models
bullPay-as-you-go
bullAllocation-pool
bullReservation-pool
T E C H N I C A L W H I T E P A P E R 1 0
VMware vCloud Director Resource Allocation Models
Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool
Figure 5 Overview vCloud Director Allocation Model
VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING
Allocation Limit
Guarantee Reservation
vCPU speed Limit on CPU virtual machine level
Table 3 Resource-Setting Taxonomy
The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both
T E C H N I C A L W H I T E P A P E R 1 1
VMware vCloud Director Resource Allocation Models
Resource Pool Resource Pool Resource Pool
VM VM VM VM VM VMVM
Pay-As-You-Go Model
Virtual MachinendashLevelSettings
Virtual Machinendash and Resource PoolndashLevel
Settings
Resource PoolndashLevelSettings
Reservation-Pool Model Allocation-Pool Model
VM VM
Figure 6 Overview of Allocation Models and vSphere Settings
Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo
NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC
Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC
The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Table 4 Provider VDC Resources
T E C H N I C A L W H I T E P A P E R 1 2
VMware vCloud Director Resource Allocation Models
Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults
VM VM VM
MemoryReservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
CPU Limit
CPU Reservation
Resource Pool
Figure 7 Pay-As-You-Go Model Default Settings
Figure 8 Pay-As-You-Go Model Default Settings User Interface
CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit
T E C H N I C A L W H I T E P A P E R 1 3
VMware vCloud Director Resource Allocation Models
EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent
Figure 9 Pay-As-You-Go Virtual Machine CPU Settings
Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface
EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface
Figure 10 Pay-As-You-Go Virtual Machine Memory Settings
T E C H N I C A L W H I T E P A P E R 1 4
VMware vCloud Director Resource Allocation Models
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool NA NA Unlimited NA Enabled
Virtual Machine 026GHz per vCPU
0 NA 100 NA
Table 5 Pay-As-You-Go Default Settings
A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent
Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
Figure 11 Reservation Pool Model Default Settings
T E C H N I C A L W H I T E P A P E R 1 5
VMware vCloud Director Resource Allocation Models
Figure 12 Reservation Pool Model Default Settings User Interface
NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)
The reservation pool model applies resource allocation settings on the resource pool only
CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level
Figure 13 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 1 0
VMware vCloud Director Resource Allocation Models
Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool
Figure 5 Overview vCloud Director Allocation Model
VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING
Allocation Limit
Guarantee Reservation
vCPU speed Limit on CPU virtual machine level
Table 3 Resource-Setting Taxonomy
The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both
T E C H N I C A L W H I T E P A P E R 1 1
VMware vCloud Director Resource Allocation Models
Resource Pool Resource Pool Resource Pool
VM VM VM VM VM VMVM
Pay-As-You-Go Model
Virtual MachinendashLevelSettings
Virtual Machinendash and Resource PoolndashLevel
Settings
Resource PoolndashLevelSettings
Reservation-Pool Model Allocation-Pool Model
VM VM
Figure 6 Overview of Allocation Models and vSphere Settings
Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo
NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC
Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC
The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Table 4 Provider VDC Resources
T E C H N I C A L W H I T E P A P E R 1 2
VMware vCloud Director Resource Allocation Models
Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults
VM VM VM
MemoryReservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
CPU Limit
CPU Reservation
Resource Pool
Figure 7 Pay-As-You-Go Model Default Settings
Figure 8 Pay-As-You-Go Model Default Settings User Interface
CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit
T E C H N I C A L W H I T E P A P E R 1 3
VMware vCloud Director Resource Allocation Models
EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent
Figure 9 Pay-As-You-Go Virtual Machine CPU Settings
Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface
EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface
Figure 10 Pay-As-You-Go Virtual Machine Memory Settings
T E C H N I C A L W H I T E P A P E R 1 4
VMware vCloud Director Resource Allocation Models
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool NA NA Unlimited NA Enabled
Virtual Machine 026GHz per vCPU
0 NA 100 NA
Table 5 Pay-As-You-Go Default Settings
A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent
Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
Figure 11 Reservation Pool Model Default Settings
T E C H N I C A L W H I T E P A P E R 1 5
VMware vCloud Director Resource Allocation Models
Figure 12 Reservation Pool Model Default Settings User Interface
NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)
The reservation pool model applies resource allocation settings on the resource pool only
CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level
Figure 13 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 1 1
VMware vCloud Director Resource Allocation Models
Resource Pool Resource Pool Resource Pool
VM VM VM VM VM VMVM
Pay-As-You-Go Model
Virtual MachinendashLevelSettings
Virtual Machinendash and Resource PoolndashLevel
Settings
Resource PoolndashLevelSettings
Reservation-Pool Model Allocation-Pool Model
VM VM
Figure 6 Overview of Allocation Models and vSphere Settings
Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo
NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC
Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC
The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Table 4 Provider VDC Resources
T E C H N I C A L W H I T E P A P E R 1 2
VMware vCloud Director Resource Allocation Models
Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults
VM VM VM
MemoryReservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
CPU Limit
CPU Reservation
Resource Pool
Figure 7 Pay-As-You-Go Model Default Settings
Figure 8 Pay-As-You-Go Model Default Settings User Interface
CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit
T E C H N I C A L W H I T E P A P E R 1 3
VMware vCloud Director Resource Allocation Models
EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent
Figure 9 Pay-As-You-Go Virtual Machine CPU Settings
Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface
EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface
Figure 10 Pay-As-You-Go Virtual Machine Memory Settings
T E C H N I C A L W H I T E P A P E R 1 4
VMware vCloud Director Resource Allocation Models
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool NA NA Unlimited NA Enabled
Virtual Machine 026GHz per vCPU
0 NA 100 NA
Table 5 Pay-As-You-Go Default Settings
A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent
Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
Figure 11 Reservation Pool Model Default Settings
T E C H N I C A L W H I T E P A P E R 1 5
VMware vCloud Director Resource Allocation Models
Figure 12 Reservation Pool Model Default Settings User Interface
NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)
The reservation pool model applies resource allocation settings on the resource pool only
CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level
Figure 13 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 1 2
VMware vCloud Director Resource Allocation Models
Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults
VM VM VM
MemoryReservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
CPU Limit
CPU Reservation
Resource Pool
Figure 7 Pay-As-You-Go Model Default Settings
Figure 8 Pay-As-You-Go Model Default Settings User Interface
CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit
T E C H N I C A L W H I T E P A P E R 1 3
VMware vCloud Director Resource Allocation Models
EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent
Figure 9 Pay-As-You-Go Virtual Machine CPU Settings
Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface
EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface
Figure 10 Pay-As-You-Go Virtual Machine Memory Settings
T E C H N I C A L W H I T E P A P E R 1 4
VMware vCloud Director Resource Allocation Models
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool NA NA Unlimited NA Enabled
Virtual Machine 026GHz per vCPU
0 NA 100 NA
Table 5 Pay-As-You-Go Default Settings
A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent
Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
Figure 11 Reservation Pool Model Default Settings
T E C H N I C A L W H I T E P A P E R 1 5
VMware vCloud Director Resource Allocation Models
Figure 12 Reservation Pool Model Default Settings User Interface
NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)
The reservation pool model applies resource allocation settings on the resource pool only
CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level
Figure 13 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 1 3
VMware vCloud Director Resource Allocation Models
EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent
Figure 9 Pay-As-You-Go Virtual Machine CPU Settings
Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface
EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface
Figure 10 Pay-As-You-Go Virtual Machine Memory Settings
T E C H N I C A L W H I T E P A P E R 1 4
VMware vCloud Director Resource Allocation Models
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool NA NA Unlimited NA Enabled
Virtual Machine 026GHz per vCPU
0 NA 100 NA
Table 5 Pay-As-You-Go Default Settings
A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent
Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
Figure 11 Reservation Pool Model Default Settings
T E C H N I C A L W H I T E P A P E R 1 5
VMware vCloud Director Resource Allocation Models
Figure 12 Reservation Pool Model Default Settings User Interface
NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)
The reservation pool model applies resource allocation settings on the resource pool only
CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level
Figure 13 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 1 4
VMware vCloud Director Resource Allocation Models
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool NA NA Unlimited NA Enabled
Virtual Machine 026GHz per vCPU
0 NA 100 NA
Table 5 Pay-As-You-Go Default Settings
A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent
Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
Figure 11 Reservation Pool Model Default Settings
T E C H N I C A L W H I T E P A P E R 1 5
VMware vCloud Director Resource Allocation Models
Figure 12 Reservation Pool Model Default Settings User Interface
NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)
The reservation pool model applies resource allocation settings on the resource pool only
CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level
Figure 13 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 1 5
VMware vCloud Director Resource Allocation Models
Figure 12 Reservation Pool Model Default Settings User Interface
NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)
The reservation pool model applies resource allocation settings on the resource pool only
CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level
Figure 13 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 1 6
VMware vCloud Director Resource Allocation Models
Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level
Figure 14 Resource PoolndashLevel Memory Resource Settings
EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled
Virtual Machine NA NA NA NA NA
Table 6 Reservation Pool Default Settings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting
Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately
Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 1 7
VMware vCloud Director Resource Allocation Models
NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation
To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab
Figure 15 Virtual Machine Custom Resource Allocation Tab
Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed
Figure 16 Allocation Pool Default Settings User Interface
The allocation pool model applies resource allocation settings on resource pool and virtual machine levels
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 1 8
VMware vCloud Director Resource Allocation Models
MemoryReservation
MemoryLimit
Resource PoolndashLevel Settings
Virtual MachinendashLevel Settings
Resource Pool
VM VM VM
MemoryReservation
MemoryLimit
CPU Limit
CPU Reservation
Figure 17 Allocation Pool Model Default Settings
CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting
Figure 18 Resource PoolndashLevel CPU Resource Settings
EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 1 9
VMware vCloud Director Resource Allocation Models
Figure 19 Virtual MachinendashLevel CPU Resource Settings
Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well
Figure 20 Resource PoolndashLevel Memory Resource Settings
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 2 0
VMware vCloud Director Resource Allocation Models
Figure 21 Virtual MachinendashLevel Memory Resource Settings
EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
EXPANDABLE RESERVATION
Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled
Virtual Machine Unlimited NA 100 100 NA
Table 7 vCPU Resource Mappings
The vCloud Director user interface does not offer the possibility to adjust this setting
Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool
Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 2 1
VMware vCloud Director Resource Allocation Models
TERM DESCRIPTION
Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value
Contention Resource demand exceeds resource supply
Table 8 Resource Management Taxonomy
Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC
Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC
The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors
bullThenumberofvirtualmachinesrunningwithintheresourcepool
bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs
bullVirtualmachinememorysizeandspecifiedallocationsettings
Organization VDC
VM VM VM
Provider VDC
VirtualMachineReservedMemory Allocated
Space byOrganization
VDC
UnallocatedldquoGrowthrdquo
Space
UnreservedMemory
Withdrawnfrom Provider
VDC
Figure 22 Resource Distribution of Pay-As-You-Go Model
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 2 2
VMware vCloud Director Resource Allocation Models
Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources
The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level
Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC
VM
Virtual MachinendashLevel Reservation
ResourcesLeft for Other
Virtual Machines or Resource Pools
Provider VDC
Organization VDC
Figure 23 Virtual MachinendashLevel Reservation Extraction
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 2 3
VMware vCloud Director Resource Allocation Models
Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer
Organization VDC
VM VM VM
Provider VDC
Distributed to VirtualMachineBased onActivity
LimitedAllocation
Unavailableto Organization
VDC
UnreservedMemory
Withdrawnfrom
ProviderVDC
Figure 24 Resource Pool Model Resource Distribution
The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority
The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation
Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC
By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case
During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 2 4
VMware vCloud Director Resource Allocation Models
An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool
Organization VDC20GB Allocation50 Guarantee
VM12GB
UnreservedMemory
10GB BurstSpace Available
10GBReservedMemoryAvailable
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 25 Admission Control Phase 1
Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 2 5
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM22GB
VM1Unreserved
Memory
10GB BurstSpace Available
9GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
Figure 26 Admission Control Phase 2
During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 2 6
VMware vCloud Director Resource Allocation Models
Organization VDC20GB Allocation50 Guarantee
VM112GB
UnreservedMemory
10GB BurstSpace Available
0GBReservedMemoryAvailable
Withdrawnfrom
Resource PoolReservation
1GBVirtual MachinendashLevel
Reservation
Admission Control
1GBOpportunistic
Access
VM1
VM2
VM3
VM4
VM5
VM6
VM7
VM8
VM9
VM10
Figure 27 Admission Control Phase 3 ndash Failure
When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation
NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell
Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 2 7
VMware vCloud Director Resource Allocation Models
Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC
40GB Allocated100 Guaranteed
ReservedMemory
UnreservedMemory
40GB Allocated100 Guaranteed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
40GB A
lloca
ted
50
Gua
rant
eed
Opp
ortu
nist
ic
Acces
s
Provider VDC80GB
Provider VDC80GB
Organization VDC
Organization VDC
OrgVDC
OrgVDC
OrgVDC
Figure 28 TBD
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 2 8
VMware vCloud Director Resource Allocation Models
vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model
Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model
bullCatalogtemplatestorage
bullTemporarytransientneeds
bullLow-costlow-performanceworkloads
Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees
Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model
bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications
bullldquoDedicatedrdquoresources
bullTheneedforeasilybudgetedbilling
These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options
Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following
bullPer-projecthosting
bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems
bullMinimumguaranteesofperformance
Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC
Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 2 9
VMware vCloud Director Resource Allocation Models
Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC
vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules
RESOURCE CONFIGURATION
CPU 1
Memory 256MB
Table 9 vShield Virtual Hardware Configuration
In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited
SETTING LIMIT RESERVATION
CPU Unlimited 64MHz
Memory 256MB 64MB
Table 10 vShield Edge Resource Allocation Settings
The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation
A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity
EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing
250 organizations with one NAT network each =
bull250vShieldEdgedevicesdeployed
bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)
bullApproximately16GHzofreservedCPUresources
NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 3 0
VMware vCloud Director Resource Allocation Models
Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented
bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC
bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC
A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11
SETTING CPU MEMORY
Provider VDC 15GHz 50GB
Virtual machine 1 2 vCPU 4096MB
Virtual machine 2 2 vCPU 4096MB
Memory overhead reservation virtual machine
250MB
Table 11 Test vCloud Environment Configuration
In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC
NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement
Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 026GHz per vCPU 0 100 100
Customer 2 026GHz per vCPU 0 100 50
Customer 3 026GHz per vCPU 0 100 0
Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview
The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 3 1
VMware vCloud Director Resource Allocation Models
Reserved Memory
17384MB 9192MB 8192MB 15342MB 1000MB
Unreserved Memory
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand
Provider VDC
17384MB Demand
Figure 29 Resource Distribution Pay-As-You-Go Scenario
Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed
Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation
Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC
The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3
bullThevirtualmachinesofcustomer1receivetheirfullallocation
ndash 17184MB of reserved resources are allocated
ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory
bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee
ndash 8192MB of reserved resources are allocated
ndash There is an additional 1000MB of memory overhead reservation
ndash The total reserved memory allocation is 9192MB
ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory
bullAfterpower-onofthevirtualmachinesofcustomer3
ndash 1000MB are withdrawn from the provider VDC unreserved memory pool
ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 3 2
VMware vCloud Director Resource Allocation Models
8192MB
50Priority
50Priority
11812MB 11812MB
8192MB16384MB 15432MB
Provider VDC
23624MB 23624MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test
The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value
During contention the two resource pools receive the same priority to access memory
bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory
bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool
bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB
In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory
Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool
Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 3 3
VMware vCloud Director Resource Allocation Models
Single Allocation Model ndash Reservation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB
Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB
Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB
Table 13 Reservation Pool Resource Pool Resource Settings Overview
As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure
Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool
Customer 1 Customer 2 Customer 3
17384MB Demand 17384MB Demand 17384MB Demand
Reserved Memory
Allocated MemoryNot Used
Unavailable Memory
20480MB 15360MB 10240MB
Provider VDC
Figure 31 Resource Distribution Reservation Pool Scenario
Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources
Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines
Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 3 4
VMware vCloud Director Resource Allocation Models
Memory OverheadReservation
VM1
4GB3560
0
2390
0
3GB
2GB
1GB
0
VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2
20480MB Available
vApp1
MemoryReclamation
Customer 1
vApp2
15360MB Available
vApp1
Customer 2
vApp2
10240MB Available
vApp1
Customer 3
vApp2
Figure 32 Distribution of Memory Resources Within Resource Pools
Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings
Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC
Single Allocation Model ndash Allocation Pool
SETTING CPU LIMIT CPU RESERVATION
MEMORY LIMIT MEMORY RESERVATION
Customer 1 3GHZ 0 20GB 20GB (100)
Customer 2 3GHZ 0 20GB 15GB (75)
Customer 3 3GHZ 0 20GB 10GB (50)
Table 14 Allocation Pool Resource Settings Overview
The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC
Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 3 5
VMware vCloud Director Resource Allocation Models
The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources
The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)
Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)
Customer 1(20GB Limit)
Customer 2(20GB Limit)
17384MB Demand
Customer 3(20GB Limit)
17384MB Demand
17384MB Demand
Provider VDC
Allocated MemoryNot Used
20480MB 15360MB 5120MB 10240MB
Figure 33 Resource Distribution Allocation Pool Scenario
Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements
Unreserved Physical Resources
5120MB
2024MB 7144MB
Provider VDC
Customer 2Demand
Customer 3Demand
Figure 34 Oversubscription of Burst Space
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 3 6
VMware vCloud Director Resource Allocation Models
The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources
EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue
1 The two resource pools will receive equal priority
2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool
3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use
2024MB
50Priority
50Priority
2560MB 2560MB
2024MB7144MB 3096MB
Provider VDC
5120MB 5120MB
Provider VDC
1 2
3
Customer 3Normal Shares
Customer 2Normal Shares
Customer 3Normal Shares
Customer 2Normal Shares
Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test
Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance
Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements
Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 3 7
VMware vCloud Director Resource Allocation Models
Default Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 20GB
Customer 2 Allocation pool 3GHz 0 20GB 100
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 100
Table 15 Multiple Allocation Default Resource Settings Overview
The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory
Customer 1
Provider Virtual Datacenter
20480MBGuaranteed
Figure 36 Organization VDC Customer 1
Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory
Customer 1 Customer 2
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
Figure 37 Organization VDC Customer 2
In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 3 8
VMware vCloud Director Resource Allocation Models
Customer 1 Customer 2 Customer 3
Provider Virtual Datacenter
20480MBGuaranteed
20480MBGuaranteed
0MBGuaranteed
Figure 38 Organization VDC Customer 3
After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory
Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB
The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool
Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible
Custom Resource Allocation Settings Test
SETTING ALLOCATION MODEL
CPU LIMIT CPU RESERVATION
MEMORY LIMIT
MEMORY RESERVATION
Customer 1 Reservation pool 3GHz 100 100 15GB
Customer 2 Allocation pool 3GHz 75 20GB 75
Customer 3 Pay-as-you-go 026GHz per vCPU
0 100 75
Table 16 Multiple Allocation Custom Resource Settings Overview
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 3 9
VMware vCloud Director Resource Allocation Models
The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent
As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory
Customer 1
Provider VDC
15360MBGuaranteed
Figure 39 Organization VDC Customer 1
Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed
Figure 40 Organization VDC Customer 2
Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 4 0
VMware vCloud Director Resource Allocation Models
Customer 1
Provider VDC
15360MBGuaranteed
Customer 2
15360MBGuaranteed Customer 3
0MBGuaranteed
Figure 41 Organization VDC Customer 3
After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC
Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC
Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore
Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs
Customer 2
Provider VDC
15360MBGuaranteed
5120MBOpportunistic
Allocation
Figure 42 Opportunistic Resource Allocations
Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
T E C H N I C A L W H I T E P A P E R 4 1
VMware vCloud Director Resource Allocation Models
In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB
Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory
In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads
NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured
About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)
bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere
bullFollowFrankonTwitterfrankdenneman
Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)
bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud
bullFollowChrisonTwitterCColotti
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01
VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01