VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models....

42
VMware vCloud ® Director Resource Allocation Models TECHNICAL MARKETING DOCUMENTATION V 1.0/UPDATED AUGUST 2012

Transcript of VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models....

Page 1: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

VMware vCloudreg Director trade

Resource Allocation ModelsT E C H N I C A L M A R K E T I N G D O C U M E N TAT I O NV 1 0 U P D AT E D A U G U S T 2 0 1 2

VMware vCloud Director Resource Allocation Models

T E C H N I C A L W H I T E P A P E R 2

Table of Contents

Purpose and Overview 4

Target Audience 4

Interpreting This Document 4

Reference Material 4

Resource Management Constructs 5

vCloud Director Constructs 5

vSphere Resource Constructs 6

vCloud Director Architecture 7

Architecting the Provider Virtual Datacenter 8

Organization Virtual Datacenters 9

Allocation Models 10

Pay-As-You-Go Model Settings 12

Expandable Reservation 14

Reservation Pool Model Settings 14

Expandable Reservation 16

Custom PerndashVirtual Machine Resource Settings 16

Allocation Pool Model Settings 17

Expandable Reservation 20

Review of Resource Distribution 20

Virtual Machine Distribution on Storage Level 21

Pay-As-You-Go Model 21

Reservation Pool Model 23

Allocation Pool Model 23

vCloud Allocation Models in Practice 28

Consumer Functional Requirements 28

Pay-As-You-Go Model 28

Reservation Pool Model 28

Allocation Pool Model 28

Catalog Organization Virtual Datacenter 29

vCloud System Virtual Datacenter 29

Test Cases 30

Single Allocation Model Tests 30

Single Allocation Model ndash Pay-As-You-Go 30

Single Allocation Model ndash Reservation Pool 33

Single Allocation Model ndash Allocation Pool 34

VMware vCloud Director Resource Allocation Models

T E C H N I C A L W H I T E P A P E R 3

Multiple Allocation Models 36

Default Resource Allocation Settings Test 37

Custom Resource Allocation Settings Test 38

About the Authors 41

T E C H N I C A L W H I T E P A P E R 4

VMware vCloud Director Resource Allocation Models

Purpose and OverviewVMware vCloudreg Directortrade enables users to build a private cloudndashbased infrastructure as a service (IaaS) offering within their organization By providing a secure on-demand ability for end users to deploy workloads companies can realize a level of agility previously thought impossible

Resource design for vCloud Director involves examining requirements to determine how best to partition and organize resources With the commoditization of infrastructure resources the ability to scale them becomes increasingly important When designing for vCloud Director keep in mind that the ultimate consumers of the product are the end users of the system Taking a top-down approach to vCloud Director design necessitates an understanding of the new abstractions introduced in the vCloud API and how they map to traditional VMware vSpherereg objects This paper was developed to provide additional insight and information as to how vCloud Director allocation models relate to vSphere resource management The paper reviews and discusses various design considerations and operational procedures

Target AudienceThe target audience of this document is individuals with a technical background who will be designing deploying or managing a vCloud Director infrastructure This includes but is not limited to technical consultants infrastructure architects IT managers implementation engineers partner engineers sales engineers and customer staff This solution brief is not intended to replace the VMware vCloud Architecture Toolkit It instead is meant to supplement knowledge and provide additional information As vCloud Director implementations have evolved there has been a noticeable increase in provider and consumer confusion about the vCloud Director allocation concepts as related to vSphere

Interpreting This DocumentThe overall structure of this case study is for the most part self-explanatory However throughout this document there are key points of particular importance that will be highlighted to the user These points will be identified with one of the following labels

bullNOTEndashGeneralpointofimportanceorafurtherexplanationwithinaparticularsection

bullEXAMPLEndashAspecificusecasetoillustrateaconceptwithinasection

The authors of this document assume that the reader is familiar with vCloud Director vSphere VMwarereg vCenter Servertrade VMware vSphere High Availability (vSphere HA) VMware vSphere Distributed Resource Schedulertrade (vSphere DRS) and VMware vSphere Storage DRStrade technology and terminology

Reference MaterialReading the following documents that have been used to develop this solution brief is recommended

bullvSphere5ResourceManagementGuide

bullVMwarevCloudArchitectureToolkitemsp

T E C H N I C A L W H I T E P A P E R 5

VMware vCloud Director Resource Allocation Models

Resource Management ConstructsvCloud Director introduces logical constructs to facilitate multitenancy and provide interoperability between vCloud Director instances built to the vCloud API standard

vCloud Director ConstructsTable 1 describes the logical constructs in vCloud Director that are used to abstract the underlying vSphere constructs

VCLOUD DIRECTOR CONSTRUCT

DESCRIPTION

Organization An organization is the unit of multitenancy that represents a single logical security boundary An organization contains users virtual datacenters and networks

Provider Virtual Datacenter A provider virtual datacenter (VDC) is a grouping of compute and storage resources from a single vCenter Server instance A provider VDC consists of a pool of physical compute resources and one or more datastores Multiple organizations can share provider VDC resources

Organization Virtual Datacenter An organization virtual datacenter is a subgrouping of compute and storage resources allocated from a provider VDC and assigned to a single organization An organization VDC is provisioned resources using vCloud Director resource allocation models These are represented in vSphere by ldquoresource poolsrdquo defined in Table 2

Resource Allocation Models Resource allocation models define how resources are provisioned to an organizationrsquos VDC from the provider VDC They also define how resources can be used when deploying virtual applications (vApps) within the organization VDC

vApp A vApp is a container for a distributed software solution and is the standard unit of deployment in vCloud Director A vCloud Director vApp is very different from a vSphere vApp in the manner it is instantiated and consumed in vCloud Director It enables power operations to be defined and specifically ordered It consists of one or more virtual machines and can be imported or exported as an OVF package A vCloud vApp can have additional vCloud-specific constructs such as vApp networks

Table 1 vCloud Director Constructs

T E C H N I C A L W H I T E P A P E R 6

VMware vCloud Director Resource Allocation Models

vSphere Resource Constructs Table 2 describes the resource management constructs in vSphere

VSPHERE RESOURCE CONSTRUCT

DESCRIPTION

Distributed Resource Scheduler Cluster

A vSphere DRS cluster is a collection of vSphere hosts with shared resources It must be created first to obtain the benefits of cluster-level resource management such as resource pools

Resource Pools A resource pool is a logical abstraction of physical resources for flexible management of resources Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources

Reservation A reservation specifies the guaranteed minimum physical resource allocation for a virtual machine or resource pool A power-on operation of a virtual machine (via vCenter Server or directly accessing the vSphere host with VMware vSphere Clienttrade) will succeed only if there are enough unreserved resources to satisfy the reservation needed by the virtual machine The server guarantees that amount even when the physical server is heavily loaded The reservation is expressed in physical units (megahertz or megabytes)

Limit Limit specifies an upper boundary for CPU or memory resources that can be allocated to a virtual machine or resource A server can allocate more than the reservation to a virtual machine but it never allocates more than the limit even if there are unused resources on the system The limit is expressed in concrete units (megahertz or megabytes) The default value for limits is ldquounlimitedrdquo When the memory value is unlimited the amount of memory configured for the virtual machine when it was created becomes its effective limit

Shares Shares specify the relative importance of a virtual machine or resource pool If a virtual machine has twice as many shares of a resource as another virtual machine it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources

Memory Overhead Reservation Memory overhead reservation is the amount of consumed memory being used for virtualization purposes Overhead memory includes space reserved for the virtual machine frame buffer and various virtualization data structures such as shadow page tables

Table 2 vSphere Resource Management Constructs

T E C H N I C A L W H I T E P A P E R 7

VMware vCloud Director Resource Allocation Models

vCloud Director ArchitectureThe vSphere infrastructure provides the foundation of the vCloud Director architecture Building a vCloud service requires an understanding of the individual constructs of the entire architecture

The vSphere layer provides a consumable set of resources available to the VMware vCloud environment and will present these resources to a provider VDC The provider VDC supplies an abstraction layer for resources As each organization VDC withdraws resources from its provider VDC the provider VDC creates the association between the organization VDC and resources at the vSphere layer The vCloud allocation model defined on each organization VDC dictates how the resources of the vSphere layer are consumed and it influences the availability of resources to the other organization VDCs sharing the same provider VDC

O

rgan

izat

ion

Virtual D

atacenter Allocation Model Provider Virtual Datacenter

vSphere R

esou

rces

ConstructInterdynamics

Figure 1 Cycle of vSphere and vCloud Construct Interdynamics

This interaction of vSphere and vCloud constructs determines the flexibility of the resource availability within the vCloud environment And eventually it is the resource availability that drives the consolidation ratio of the organization VDCs within a provider VDC The remaining chapters expand on the architecture of the provider VDC and how vCloud allocation models consume resources

T E C H N I C A L W H I T E P A P E R 8

VMware vCloud Director Resource Allocation Models

Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores

Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering

EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC

Cluster

Storage

Datastores

Provider Virtual Datacenter

Compute

ResourcePool

Figure 2 Provider VDC Resource Mapping

A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster

Provider VDC

Resource Pool

DRS Cluster

Map to

Figure 3 Mapping Provider VDC to Cluster or Resource Pool

By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool

The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources

T E C H N I C A L W H I T E P A P E R 9

VMware vCloud Director Resource Allocation Models

NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible

vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool

Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources

An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations

Organization VDC

Organization Customer 1

GoldProvider

BronzeProvider

Provider VDC

Organization VDC

Organization VDC

Provider VDC

Figure 4 Multiple Organization VDCs

Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models

bullPay-as-you-go

bullAllocation-pool

bullReservation-pool

T E C H N I C A L W H I T E P A P E R 1 0

VMware vCloud Director Resource Allocation Models

Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool

Figure 5 Overview vCloud Director Allocation Model

VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING

Allocation Limit

Guarantee Reservation

vCPU speed Limit on CPU virtual machine level

Table 3 Resource-Setting Taxonomy

The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both

T E C H N I C A L W H I T E P A P E R 1 1

VMware vCloud Director Resource Allocation Models

Resource Pool Resource Pool Resource Pool

VM VM VM VM VM VMVM

Pay-As-You-Go Model

Virtual MachinendashLevelSettings

Virtual Machinendash and Resource PoolndashLevel

Settings

Resource PoolndashLevelSettings

Reservation-Pool Model Allocation-Pool Model

VM VM

Figure 6 Overview of Allocation Models and vSphere Settings

Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo

NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC

Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC

The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Table 4 Provider VDC Resources

T E C H N I C A L W H I T E P A P E R 1 2

VMware vCloud Director Resource Allocation Models

Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults

VM VM VM

MemoryReservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

CPU Limit

CPU Reservation

Resource Pool

Figure 7 Pay-As-You-Go Model Default Settings

Figure 8 Pay-As-You-Go Model Default Settings User Interface

CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit

T E C H N I C A L W H I T E P A P E R 1 3

VMware vCloud Director Resource Allocation Models

EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent

Figure 9 Pay-As-You-Go Virtual Machine CPU Settings

Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface

EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface

Figure 10 Pay-As-You-Go Virtual Machine Memory Settings

T E C H N I C A L W H I T E P A P E R 1 4

VMware vCloud Director Resource Allocation Models

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool NA NA Unlimited NA Enabled

Virtual Machine 026GHz per vCPU

0 NA 100 NA

Table 5 Pay-As-You-Go Default Settings

A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent

Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

Figure 11 Reservation Pool Model Default Settings

T E C H N I C A L W H I T E P A P E R 1 5

VMware vCloud Director Resource Allocation Models

Figure 12 Reservation Pool Model Default Settings User Interface

NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)

The reservation pool model applies resource allocation settings on the resource pool only

CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level

Figure 13 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 2: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

VMware vCloud Director Resource Allocation Models

T E C H N I C A L W H I T E P A P E R 2

Table of Contents

Purpose and Overview 4

Target Audience 4

Interpreting This Document 4

Reference Material 4

Resource Management Constructs 5

vCloud Director Constructs 5

vSphere Resource Constructs 6

vCloud Director Architecture 7

Architecting the Provider Virtual Datacenter 8

Organization Virtual Datacenters 9

Allocation Models 10

Pay-As-You-Go Model Settings 12

Expandable Reservation 14

Reservation Pool Model Settings 14

Expandable Reservation 16

Custom PerndashVirtual Machine Resource Settings 16

Allocation Pool Model Settings 17

Expandable Reservation 20

Review of Resource Distribution 20

Virtual Machine Distribution on Storage Level 21

Pay-As-You-Go Model 21

Reservation Pool Model 23

Allocation Pool Model 23

vCloud Allocation Models in Practice 28

Consumer Functional Requirements 28

Pay-As-You-Go Model 28

Reservation Pool Model 28

Allocation Pool Model 28

Catalog Organization Virtual Datacenter 29

vCloud System Virtual Datacenter 29

Test Cases 30

Single Allocation Model Tests 30

Single Allocation Model ndash Pay-As-You-Go 30

Single Allocation Model ndash Reservation Pool 33

Single Allocation Model ndash Allocation Pool 34

VMware vCloud Director Resource Allocation Models

T E C H N I C A L W H I T E P A P E R 3

Multiple Allocation Models 36

Default Resource Allocation Settings Test 37

Custom Resource Allocation Settings Test 38

About the Authors 41

T E C H N I C A L W H I T E P A P E R 4

VMware vCloud Director Resource Allocation Models

Purpose and OverviewVMware vCloudreg Directortrade enables users to build a private cloudndashbased infrastructure as a service (IaaS) offering within their organization By providing a secure on-demand ability for end users to deploy workloads companies can realize a level of agility previously thought impossible

Resource design for vCloud Director involves examining requirements to determine how best to partition and organize resources With the commoditization of infrastructure resources the ability to scale them becomes increasingly important When designing for vCloud Director keep in mind that the ultimate consumers of the product are the end users of the system Taking a top-down approach to vCloud Director design necessitates an understanding of the new abstractions introduced in the vCloud API and how they map to traditional VMware vSpherereg objects This paper was developed to provide additional insight and information as to how vCloud Director allocation models relate to vSphere resource management The paper reviews and discusses various design considerations and operational procedures

Target AudienceThe target audience of this document is individuals with a technical background who will be designing deploying or managing a vCloud Director infrastructure This includes but is not limited to technical consultants infrastructure architects IT managers implementation engineers partner engineers sales engineers and customer staff This solution brief is not intended to replace the VMware vCloud Architecture Toolkit It instead is meant to supplement knowledge and provide additional information As vCloud Director implementations have evolved there has been a noticeable increase in provider and consumer confusion about the vCloud Director allocation concepts as related to vSphere

Interpreting This DocumentThe overall structure of this case study is for the most part self-explanatory However throughout this document there are key points of particular importance that will be highlighted to the user These points will be identified with one of the following labels

bullNOTEndashGeneralpointofimportanceorafurtherexplanationwithinaparticularsection

bullEXAMPLEndashAspecificusecasetoillustrateaconceptwithinasection

The authors of this document assume that the reader is familiar with vCloud Director vSphere VMwarereg vCenter Servertrade VMware vSphere High Availability (vSphere HA) VMware vSphere Distributed Resource Schedulertrade (vSphere DRS) and VMware vSphere Storage DRStrade technology and terminology

Reference MaterialReading the following documents that have been used to develop this solution brief is recommended

bullvSphere5ResourceManagementGuide

bullVMwarevCloudArchitectureToolkitemsp

T E C H N I C A L W H I T E P A P E R 5

VMware vCloud Director Resource Allocation Models

Resource Management ConstructsvCloud Director introduces logical constructs to facilitate multitenancy and provide interoperability between vCloud Director instances built to the vCloud API standard

vCloud Director ConstructsTable 1 describes the logical constructs in vCloud Director that are used to abstract the underlying vSphere constructs

VCLOUD DIRECTOR CONSTRUCT

DESCRIPTION

Organization An organization is the unit of multitenancy that represents a single logical security boundary An organization contains users virtual datacenters and networks

Provider Virtual Datacenter A provider virtual datacenter (VDC) is a grouping of compute and storage resources from a single vCenter Server instance A provider VDC consists of a pool of physical compute resources and one or more datastores Multiple organizations can share provider VDC resources

Organization Virtual Datacenter An organization virtual datacenter is a subgrouping of compute and storage resources allocated from a provider VDC and assigned to a single organization An organization VDC is provisioned resources using vCloud Director resource allocation models These are represented in vSphere by ldquoresource poolsrdquo defined in Table 2

Resource Allocation Models Resource allocation models define how resources are provisioned to an organizationrsquos VDC from the provider VDC They also define how resources can be used when deploying virtual applications (vApps) within the organization VDC

vApp A vApp is a container for a distributed software solution and is the standard unit of deployment in vCloud Director A vCloud Director vApp is very different from a vSphere vApp in the manner it is instantiated and consumed in vCloud Director It enables power operations to be defined and specifically ordered It consists of one or more virtual machines and can be imported or exported as an OVF package A vCloud vApp can have additional vCloud-specific constructs such as vApp networks

Table 1 vCloud Director Constructs

T E C H N I C A L W H I T E P A P E R 6

VMware vCloud Director Resource Allocation Models

vSphere Resource Constructs Table 2 describes the resource management constructs in vSphere

VSPHERE RESOURCE CONSTRUCT

DESCRIPTION

Distributed Resource Scheduler Cluster

A vSphere DRS cluster is a collection of vSphere hosts with shared resources It must be created first to obtain the benefits of cluster-level resource management such as resource pools

Resource Pools A resource pool is a logical abstraction of physical resources for flexible management of resources Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources

Reservation A reservation specifies the guaranteed minimum physical resource allocation for a virtual machine or resource pool A power-on operation of a virtual machine (via vCenter Server or directly accessing the vSphere host with VMware vSphere Clienttrade) will succeed only if there are enough unreserved resources to satisfy the reservation needed by the virtual machine The server guarantees that amount even when the physical server is heavily loaded The reservation is expressed in physical units (megahertz or megabytes)

Limit Limit specifies an upper boundary for CPU or memory resources that can be allocated to a virtual machine or resource A server can allocate more than the reservation to a virtual machine but it never allocates more than the limit even if there are unused resources on the system The limit is expressed in concrete units (megahertz or megabytes) The default value for limits is ldquounlimitedrdquo When the memory value is unlimited the amount of memory configured for the virtual machine when it was created becomes its effective limit

Shares Shares specify the relative importance of a virtual machine or resource pool If a virtual machine has twice as many shares of a resource as another virtual machine it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources

Memory Overhead Reservation Memory overhead reservation is the amount of consumed memory being used for virtualization purposes Overhead memory includes space reserved for the virtual machine frame buffer and various virtualization data structures such as shadow page tables

Table 2 vSphere Resource Management Constructs

T E C H N I C A L W H I T E P A P E R 7

VMware vCloud Director Resource Allocation Models

vCloud Director ArchitectureThe vSphere infrastructure provides the foundation of the vCloud Director architecture Building a vCloud service requires an understanding of the individual constructs of the entire architecture

The vSphere layer provides a consumable set of resources available to the VMware vCloud environment and will present these resources to a provider VDC The provider VDC supplies an abstraction layer for resources As each organization VDC withdraws resources from its provider VDC the provider VDC creates the association between the organization VDC and resources at the vSphere layer The vCloud allocation model defined on each organization VDC dictates how the resources of the vSphere layer are consumed and it influences the availability of resources to the other organization VDCs sharing the same provider VDC

O

rgan

izat

ion

Virtual D

atacenter Allocation Model Provider Virtual Datacenter

vSphere R

esou

rces

ConstructInterdynamics

Figure 1 Cycle of vSphere and vCloud Construct Interdynamics

This interaction of vSphere and vCloud constructs determines the flexibility of the resource availability within the vCloud environment And eventually it is the resource availability that drives the consolidation ratio of the organization VDCs within a provider VDC The remaining chapters expand on the architecture of the provider VDC and how vCloud allocation models consume resources

T E C H N I C A L W H I T E P A P E R 8

VMware vCloud Director Resource Allocation Models

Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores

Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering

EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC

Cluster

Storage

Datastores

Provider Virtual Datacenter

Compute

ResourcePool

Figure 2 Provider VDC Resource Mapping

A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster

Provider VDC

Resource Pool

DRS Cluster

Map to

Figure 3 Mapping Provider VDC to Cluster or Resource Pool

By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool

The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources

T E C H N I C A L W H I T E P A P E R 9

VMware vCloud Director Resource Allocation Models

NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible

vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool

Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources

An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations

Organization VDC

Organization Customer 1

GoldProvider

BronzeProvider

Provider VDC

Organization VDC

Organization VDC

Provider VDC

Figure 4 Multiple Organization VDCs

Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models

bullPay-as-you-go

bullAllocation-pool

bullReservation-pool

T E C H N I C A L W H I T E P A P E R 1 0

VMware vCloud Director Resource Allocation Models

Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool

Figure 5 Overview vCloud Director Allocation Model

VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING

Allocation Limit

Guarantee Reservation

vCPU speed Limit on CPU virtual machine level

Table 3 Resource-Setting Taxonomy

The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both

T E C H N I C A L W H I T E P A P E R 1 1

VMware vCloud Director Resource Allocation Models

Resource Pool Resource Pool Resource Pool

VM VM VM VM VM VMVM

Pay-As-You-Go Model

Virtual MachinendashLevelSettings

Virtual Machinendash and Resource PoolndashLevel

Settings

Resource PoolndashLevelSettings

Reservation-Pool Model Allocation-Pool Model

VM VM

Figure 6 Overview of Allocation Models and vSphere Settings

Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo

NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC

Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC

The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Table 4 Provider VDC Resources

T E C H N I C A L W H I T E P A P E R 1 2

VMware vCloud Director Resource Allocation Models

Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults

VM VM VM

MemoryReservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

CPU Limit

CPU Reservation

Resource Pool

Figure 7 Pay-As-You-Go Model Default Settings

Figure 8 Pay-As-You-Go Model Default Settings User Interface

CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit

T E C H N I C A L W H I T E P A P E R 1 3

VMware vCloud Director Resource Allocation Models

EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent

Figure 9 Pay-As-You-Go Virtual Machine CPU Settings

Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface

EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface

Figure 10 Pay-As-You-Go Virtual Machine Memory Settings

T E C H N I C A L W H I T E P A P E R 1 4

VMware vCloud Director Resource Allocation Models

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool NA NA Unlimited NA Enabled

Virtual Machine 026GHz per vCPU

0 NA 100 NA

Table 5 Pay-As-You-Go Default Settings

A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent

Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

Figure 11 Reservation Pool Model Default Settings

T E C H N I C A L W H I T E P A P E R 1 5

VMware vCloud Director Resource Allocation Models

Figure 12 Reservation Pool Model Default Settings User Interface

NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)

The reservation pool model applies resource allocation settings on the resource pool only

CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level

Figure 13 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 3: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

VMware vCloud Director Resource Allocation Models

T E C H N I C A L W H I T E P A P E R 3

Multiple Allocation Models 36

Default Resource Allocation Settings Test 37

Custom Resource Allocation Settings Test 38

About the Authors 41

T E C H N I C A L W H I T E P A P E R 4

VMware vCloud Director Resource Allocation Models

Purpose and OverviewVMware vCloudreg Directortrade enables users to build a private cloudndashbased infrastructure as a service (IaaS) offering within their organization By providing a secure on-demand ability for end users to deploy workloads companies can realize a level of agility previously thought impossible

Resource design for vCloud Director involves examining requirements to determine how best to partition and organize resources With the commoditization of infrastructure resources the ability to scale them becomes increasingly important When designing for vCloud Director keep in mind that the ultimate consumers of the product are the end users of the system Taking a top-down approach to vCloud Director design necessitates an understanding of the new abstractions introduced in the vCloud API and how they map to traditional VMware vSpherereg objects This paper was developed to provide additional insight and information as to how vCloud Director allocation models relate to vSphere resource management The paper reviews and discusses various design considerations and operational procedures

Target AudienceThe target audience of this document is individuals with a technical background who will be designing deploying or managing a vCloud Director infrastructure This includes but is not limited to technical consultants infrastructure architects IT managers implementation engineers partner engineers sales engineers and customer staff This solution brief is not intended to replace the VMware vCloud Architecture Toolkit It instead is meant to supplement knowledge and provide additional information As vCloud Director implementations have evolved there has been a noticeable increase in provider and consumer confusion about the vCloud Director allocation concepts as related to vSphere

Interpreting This DocumentThe overall structure of this case study is for the most part self-explanatory However throughout this document there are key points of particular importance that will be highlighted to the user These points will be identified with one of the following labels

bullNOTEndashGeneralpointofimportanceorafurtherexplanationwithinaparticularsection

bullEXAMPLEndashAspecificusecasetoillustrateaconceptwithinasection

The authors of this document assume that the reader is familiar with vCloud Director vSphere VMwarereg vCenter Servertrade VMware vSphere High Availability (vSphere HA) VMware vSphere Distributed Resource Schedulertrade (vSphere DRS) and VMware vSphere Storage DRStrade technology and terminology

Reference MaterialReading the following documents that have been used to develop this solution brief is recommended

bullvSphere5ResourceManagementGuide

bullVMwarevCloudArchitectureToolkitemsp

T E C H N I C A L W H I T E P A P E R 5

VMware vCloud Director Resource Allocation Models

Resource Management ConstructsvCloud Director introduces logical constructs to facilitate multitenancy and provide interoperability between vCloud Director instances built to the vCloud API standard

vCloud Director ConstructsTable 1 describes the logical constructs in vCloud Director that are used to abstract the underlying vSphere constructs

VCLOUD DIRECTOR CONSTRUCT

DESCRIPTION

Organization An organization is the unit of multitenancy that represents a single logical security boundary An organization contains users virtual datacenters and networks

Provider Virtual Datacenter A provider virtual datacenter (VDC) is a grouping of compute and storage resources from a single vCenter Server instance A provider VDC consists of a pool of physical compute resources and one or more datastores Multiple organizations can share provider VDC resources

Organization Virtual Datacenter An organization virtual datacenter is a subgrouping of compute and storage resources allocated from a provider VDC and assigned to a single organization An organization VDC is provisioned resources using vCloud Director resource allocation models These are represented in vSphere by ldquoresource poolsrdquo defined in Table 2

Resource Allocation Models Resource allocation models define how resources are provisioned to an organizationrsquos VDC from the provider VDC They also define how resources can be used when deploying virtual applications (vApps) within the organization VDC

vApp A vApp is a container for a distributed software solution and is the standard unit of deployment in vCloud Director A vCloud Director vApp is very different from a vSphere vApp in the manner it is instantiated and consumed in vCloud Director It enables power operations to be defined and specifically ordered It consists of one or more virtual machines and can be imported or exported as an OVF package A vCloud vApp can have additional vCloud-specific constructs such as vApp networks

Table 1 vCloud Director Constructs

T E C H N I C A L W H I T E P A P E R 6

VMware vCloud Director Resource Allocation Models

vSphere Resource Constructs Table 2 describes the resource management constructs in vSphere

VSPHERE RESOURCE CONSTRUCT

DESCRIPTION

Distributed Resource Scheduler Cluster

A vSphere DRS cluster is a collection of vSphere hosts with shared resources It must be created first to obtain the benefits of cluster-level resource management such as resource pools

Resource Pools A resource pool is a logical abstraction of physical resources for flexible management of resources Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources

Reservation A reservation specifies the guaranteed minimum physical resource allocation for a virtual machine or resource pool A power-on operation of a virtual machine (via vCenter Server or directly accessing the vSphere host with VMware vSphere Clienttrade) will succeed only if there are enough unreserved resources to satisfy the reservation needed by the virtual machine The server guarantees that amount even when the physical server is heavily loaded The reservation is expressed in physical units (megahertz or megabytes)

Limit Limit specifies an upper boundary for CPU or memory resources that can be allocated to a virtual machine or resource A server can allocate more than the reservation to a virtual machine but it never allocates more than the limit even if there are unused resources on the system The limit is expressed in concrete units (megahertz or megabytes) The default value for limits is ldquounlimitedrdquo When the memory value is unlimited the amount of memory configured for the virtual machine when it was created becomes its effective limit

Shares Shares specify the relative importance of a virtual machine or resource pool If a virtual machine has twice as many shares of a resource as another virtual machine it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources

Memory Overhead Reservation Memory overhead reservation is the amount of consumed memory being used for virtualization purposes Overhead memory includes space reserved for the virtual machine frame buffer and various virtualization data structures such as shadow page tables

Table 2 vSphere Resource Management Constructs

T E C H N I C A L W H I T E P A P E R 7

VMware vCloud Director Resource Allocation Models

vCloud Director ArchitectureThe vSphere infrastructure provides the foundation of the vCloud Director architecture Building a vCloud service requires an understanding of the individual constructs of the entire architecture

The vSphere layer provides a consumable set of resources available to the VMware vCloud environment and will present these resources to a provider VDC The provider VDC supplies an abstraction layer for resources As each organization VDC withdraws resources from its provider VDC the provider VDC creates the association between the organization VDC and resources at the vSphere layer The vCloud allocation model defined on each organization VDC dictates how the resources of the vSphere layer are consumed and it influences the availability of resources to the other organization VDCs sharing the same provider VDC

O

rgan

izat

ion

Virtual D

atacenter Allocation Model Provider Virtual Datacenter

vSphere R

esou

rces

ConstructInterdynamics

Figure 1 Cycle of vSphere and vCloud Construct Interdynamics

This interaction of vSphere and vCloud constructs determines the flexibility of the resource availability within the vCloud environment And eventually it is the resource availability that drives the consolidation ratio of the organization VDCs within a provider VDC The remaining chapters expand on the architecture of the provider VDC and how vCloud allocation models consume resources

T E C H N I C A L W H I T E P A P E R 8

VMware vCloud Director Resource Allocation Models

Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores

Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering

EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC

Cluster

Storage

Datastores

Provider Virtual Datacenter

Compute

ResourcePool

Figure 2 Provider VDC Resource Mapping

A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster

Provider VDC

Resource Pool

DRS Cluster

Map to

Figure 3 Mapping Provider VDC to Cluster or Resource Pool

By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool

The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources

T E C H N I C A L W H I T E P A P E R 9

VMware vCloud Director Resource Allocation Models

NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible

vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool

Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources

An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations

Organization VDC

Organization Customer 1

GoldProvider

BronzeProvider

Provider VDC

Organization VDC

Organization VDC

Provider VDC

Figure 4 Multiple Organization VDCs

Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models

bullPay-as-you-go

bullAllocation-pool

bullReservation-pool

T E C H N I C A L W H I T E P A P E R 1 0

VMware vCloud Director Resource Allocation Models

Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool

Figure 5 Overview vCloud Director Allocation Model

VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING

Allocation Limit

Guarantee Reservation

vCPU speed Limit on CPU virtual machine level

Table 3 Resource-Setting Taxonomy

The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both

T E C H N I C A L W H I T E P A P E R 1 1

VMware vCloud Director Resource Allocation Models

Resource Pool Resource Pool Resource Pool

VM VM VM VM VM VMVM

Pay-As-You-Go Model

Virtual MachinendashLevelSettings

Virtual Machinendash and Resource PoolndashLevel

Settings

Resource PoolndashLevelSettings

Reservation-Pool Model Allocation-Pool Model

VM VM

Figure 6 Overview of Allocation Models and vSphere Settings

Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo

NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC

Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC

The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Table 4 Provider VDC Resources

T E C H N I C A L W H I T E P A P E R 1 2

VMware vCloud Director Resource Allocation Models

Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults

VM VM VM

MemoryReservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

CPU Limit

CPU Reservation

Resource Pool

Figure 7 Pay-As-You-Go Model Default Settings

Figure 8 Pay-As-You-Go Model Default Settings User Interface

CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit

T E C H N I C A L W H I T E P A P E R 1 3

VMware vCloud Director Resource Allocation Models

EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent

Figure 9 Pay-As-You-Go Virtual Machine CPU Settings

Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface

EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface

Figure 10 Pay-As-You-Go Virtual Machine Memory Settings

T E C H N I C A L W H I T E P A P E R 1 4

VMware vCloud Director Resource Allocation Models

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool NA NA Unlimited NA Enabled

Virtual Machine 026GHz per vCPU

0 NA 100 NA

Table 5 Pay-As-You-Go Default Settings

A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent

Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

Figure 11 Reservation Pool Model Default Settings

T E C H N I C A L W H I T E P A P E R 1 5

VMware vCloud Director Resource Allocation Models

Figure 12 Reservation Pool Model Default Settings User Interface

NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)

The reservation pool model applies resource allocation settings on the resource pool only

CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level

Figure 13 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 4: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 4

VMware vCloud Director Resource Allocation Models

Purpose and OverviewVMware vCloudreg Directortrade enables users to build a private cloudndashbased infrastructure as a service (IaaS) offering within their organization By providing a secure on-demand ability for end users to deploy workloads companies can realize a level of agility previously thought impossible

Resource design for vCloud Director involves examining requirements to determine how best to partition and organize resources With the commoditization of infrastructure resources the ability to scale them becomes increasingly important When designing for vCloud Director keep in mind that the ultimate consumers of the product are the end users of the system Taking a top-down approach to vCloud Director design necessitates an understanding of the new abstractions introduced in the vCloud API and how they map to traditional VMware vSpherereg objects This paper was developed to provide additional insight and information as to how vCloud Director allocation models relate to vSphere resource management The paper reviews and discusses various design considerations and operational procedures

Target AudienceThe target audience of this document is individuals with a technical background who will be designing deploying or managing a vCloud Director infrastructure This includes but is not limited to technical consultants infrastructure architects IT managers implementation engineers partner engineers sales engineers and customer staff This solution brief is not intended to replace the VMware vCloud Architecture Toolkit It instead is meant to supplement knowledge and provide additional information As vCloud Director implementations have evolved there has been a noticeable increase in provider and consumer confusion about the vCloud Director allocation concepts as related to vSphere

Interpreting This DocumentThe overall structure of this case study is for the most part self-explanatory However throughout this document there are key points of particular importance that will be highlighted to the user These points will be identified with one of the following labels

bullNOTEndashGeneralpointofimportanceorafurtherexplanationwithinaparticularsection

bullEXAMPLEndashAspecificusecasetoillustrateaconceptwithinasection

The authors of this document assume that the reader is familiar with vCloud Director vSphere VMwarereg vCenter Servertrade VMware vSphere High Availability (vSphere HA) VMware vSphere Distributed Resource Schedulertrade (vSphere DRS) and VMware vSphere Storage DRStrade technology and terminology

Reference MaterialReading the following documents that have been used to develop this solution brief is recommended

bullvSphere5ResourceManagementGuide

bullVMwarevCloudArchitectureToolkitemsp

T E C H N I C A L W H I T E P A P E R 5

VMware vCloud Director Resource Allocation Models

Resource Management ConstructsvCloud Director introduces logical constructs to facilitate multitenancy and provide interoperability between vCloud Director instances built to the vCloud API standard

vCloud Director ConstructsTable 1 describes the logical constructs in vCloud Director that are used to abstract the underlying vSphere constructs

VCLOUD DIRECTOR CONSTRUCT

DESCRIPTION

Organization An organization is the unit of multitenancy that represents a single logical security boundary An organization contains users virtual datacenters and networks

Provider Virtual Datacenter A provider virtual datacenter (VDC) is a grouping of compute and storage resources from a single vCenter Server instance A provider VDC consists of a pool of physical compute resources and one or more datastores Multiple organizations can share provider VDC resources

Organization Virtual Datacenter An organization virtual datacenter is a subgrouping of compute and storage resources allocated from a provider VDC and assigned to a single organization An organization VDC is provisioned resources using vCloud Director resource allocation models These are represented in vSphere by ldquoresource poolsrdquo defined in Table 2

Resource Allocation Models Resource allocation models define how resources are provisioned to an organizationrsquos VDC from the provider VDC They also define how resources can be used when deploying virtual applications (vApps) within the organization VDC

vApp A vApp is a container for a distributed software solution and is the standard unit of deployment in vCloud Director A vCloud Director vApp is very different from a vSphere vApp in the manner it is instantiated and consumed in vCloud Director It enables power operations to be defined and specifically ordered It consists of one or more virtual machines and can be imported or exported as an OVF package A vCloud vApp can have additional vCloud-specific constructs such as vApp networks

Table 1 vCloud Director Constructs

T E C H N I C A L W H I T E P A P E R 6

VMware vCloud Director Resource Allocation Models

vSphere Resource Constructs Table 2 describes the resource management constructs in vSphere

VSPHERE RESOURCE CONSTRUCT

DESCRIPTION

Distributed Resource Scheduler Cluster

A vSphere DRS cluster is a collection of vSphere hosts with shared resources It must be created first to obtain the benefits of cluster-level resource management such as resource pools

Resource Pools A resource pool is a logical abstraction of physical resources for flexible management of resources Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources

Reservation A reservation specifies the guaranteed minimum physical resource allocation for a virtual machine or resource pool A power-on operation of a virtual machine (via vCenter Server or directly accessing the vSphere host with VMware vSphere Clienttrade) will succeed only if there are enough unreserved resources to satisfy the reservation needed by the virtual machine The server guarantees that amount even when the physical server is heavily loaded The reservation is expressed in physical units (megahertz or megabytes)

Limit Limit specifies an upper boundary for CPU or memory resources that can be allocated to a virtual machine or resource A server can allocate more than the reservation to a virtual machine but it never allocates more than the limit even if there are unused resources on the system The limit is expressed in concrete units (megahertz or megabytes) The default value for limits is ldquounlimitedrdquo When the memory value is unlimited the amount of memory configured for the virtual machine when it was created becomes its effective limit

Shares Shares specify the relative importance of a virtual machine or resource pool If a virtual machine has twice as many shares of a resource as another virtual machine it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources

Memory Overhead Reservation Memory overhead reservation is the amount of consumed memory being used for virtualization purposes Overhead memory includes space reserved for the virtual machine frame buffer and various virtualization data structures such as shadow page tables

Table 2 vSphere Resource Management Constructs

T E C H N I C A L W H I T E P A P E R 7

VMware vCloud Director Resource Allocation Models

vCloud Director ArchitectureThe vSphere infrastructure provides the foundation of the vCloud Director architecture Building a vCloud service requires an understanding of the individual constructs of the entire architecture

The vSphere layer provides a consumable set of resources available to the VMware vCloud environment and will present these resources to a provider VDC The provider VDC supplies an abstraction layer for resources As each organization VDC withdraws resources from its provider VDC the provider VDC creates the association between the organization VDC and resources at the vSphere layer The vCloud allocation model defined on each organization VDC dictates how the resources of the vSphere layer are consumed and it influences the availability of resources to the other organization VDCs sharing the same provider VDC

O

rgan

izat

ion

Virtual D

atacenter Allocation Model Provider Virtual Datacenter

vSphere R

esou

rces

ConstructInterdynamics

Figure 1 Cycle of vSphere and vCloud Construct Interdynamics

This interaction of vSphere and vCloud constructs determines the flexibility of the resource availability within the vCloud environment And eventually it is the resource availability that drives the consolidation ratio of the organization VDCs within a provider VDC The remaining chapters expand on the architecture of the provider VDC and how vCloud allocation models consume resources

T E C H N I C A L W H I T E P A P E R 8

VMware vCloud Director Resource Allocation Models

Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores

Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering

EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC

Cluster

Storage

Datastores

Provider Virtual Datacenter

Compute

ResourcePool

Figure 2 Provider VDC Resource Mapping

A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster

Provider VDC

Resource Pool

DRS Cluster

Map to

Figure 3 Mapping Provider VDC to Cluster or Resource Pool

By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool

The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources

T E C H N I C A L W H I T E P A P E R 9

VMware vCloud Director Resource Allocation Models

NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible

vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool

Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources

An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations

Organization VDC

Organization Customer 1

GoldProvider

BronzeProvider

Provider VDC

Organization VDC

Organization VDC

Provider VDC

Figure 4 Multiple Organization VDCs

Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models

bullPay-as-you-go

bullAllocation-pool

bullReservation-pool

T E C H N I C A L W H I T E P A P E R 1 0

VMware vCloud Director Resource Allocation Models

Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool

Figure 5 Overview vCloud Director Allocation Model

VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING

Allocation Limit

Guarantee Reservation

vCPU speed Limit on CPU virtual machine level

Table 3 Resource-Setting Taxonomy

The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both

T E C H N I C A L W H I T E P A P E R 1 1

VMware vCloud Director Resource Allocation Models

Resource Pool Resource Pool Resource Pool

VM VM VM VM VM VMVM

Pay-As-You-Go Model

Virtual MachinendashLevelSettings

Virtual Machinendash and Resource PoolndashLevel

Settings

Resource PoolndashLevelSettings

Reservation-Pool Model Allocation-Pool Model

VM VM

Figure 6 Overview of Allocation Models and vSphere Settings

Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo

NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC

Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC

The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Table 4 Provider VDC Resources

T E C H N I C A L W H I T E P A P E R 1 2

VMware vCloud Director Resource Allocation Models

Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults

VM VM VM

MemoryReservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

CPU Limit

CPU Reservation

Resource Pool

Figure 7 Pay-As-You-Go Model Default Settings

Figure 8 Pay-As-You-Go Model Default Settings User Interface

CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit

T E C H N I C A L W H I T E P A P E R 1 3

VMware vCloud Director Resource Allocation Models

EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent

Figure 9 Pay-As-You-Go Virtual Machine CPU Settings

Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface

EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface

Figure 10 Pay-As-You-Go Virtual Machine Memory Settings

T E C H N I C A L W H I T E P A P E R 1 4

VMware vCloud Director Resource Allocation Models

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool NA NA Unlimited NA Enabled

Virtual Machine 026GHz per vCPU

0 NA 100 NA

Table 5 Pay-As-You-Go Default Settings

A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent

Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

Figure 11 Reservation Pool Model Default Settings

T E C H N I C A L W H I T E P A P E R 1 5

VMware vCloud Director Resource Allocation Models

Figure 12 Reservation Pool Model Default Settings User Interface

NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)

The reservation pool model applies resource allocation settings on the resource pool only

CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level

Figure 13 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 5: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 5

VMware vCloud Director Resource Allocation Models

Resource Management ConstructsvCloud Director introduces logical constructs to facilitate multitenancy and provide interoperability between vCloud Director instances built to the vCloud API standard

vCloud Director ConstructsTable 1 describes the logical constructs in vCloud Director that are used to abstract the underlying vSphere constructs

VCLOUD DIRECTOR CONSTRUCT

DESCRIPTION

Organization An organization is the unit of multitenancy that represents a single logical security boundary An organization contains users virtual datacenters and networks

Provider Virtual Datacenter A provider virtual datacenter (VDC) is a grouping of compute and storage resources from a single vCenter Server instance A provider VDC consists of a pool of physical compute resources and one or more datastores Multiple organizations can share provider VDC resources

Organization Virtual Datacenter An organization virtual datacenter is a subgrouping of compute and storage resources allocated from a provider VDC and assigned to a single organization An organization VDC is provisioned resources using vCloud Director resource allocation models These are represented in vSphere by ldquoresource poolsrdquo defined in Table 2

Resource Allocation Models Resource allocation models define how resources are provisioned to an organizationrsquos VDC from the provider VDC They also define how resources can be used when deploying virtual applications (vApps) within the organization VDC

vApp A vApp is a container for a distributed software solution and is the standard unit of deployment in vCloud Director A vCloud Director vApp is very different from a vSphere vApp in the manner it is instantiated and consumed in vCloud Director It enables power operations to be defined and specifically ordered It consists of one or more virtual machines and can be imported or exported as an OVF package A vCloud vApp can have additional vCloud-specific constructs such as vApp networks

Table 1 vCloud Director Constructs

T E C H N I C A L W H I T E P A P E R 6

VMware vCloud Director Resource Allocation Models

vSphere Resource Constructs Table 2 describes the resource management constructs in vSphere

VSPHERE RESOURCE CONSTRUCT

DESCRIPTION

Distributed Resource Scheduler Cluster

A vSphere DRS cluster is a collection of vSphere hosts with shared resources It must be created first to obtain the benefits of cluster-level resource management such as resource pools

Resource Pools A resource pool is a logical abstraction of physical resources for flexible management of resources Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources

Reservation A reservation specifies the guaranteed minimum physical resource allocation for a virtual machine or resource pool A power-on operation of a virtual machine (via vCenter Server or directly accessing the vSphere host with VMware vSphere Clienttrade) will succeed only if there are enough unreserved resources to satisfy the reservation needed by the virtual machine The server guarantees that amount even when the physical server is heavily loaded The reservation is expressed in physical units (megahertz or megabytes)

Limit Limit specifies an upper boundary for CPU or memory resources that can be allocated to a virtual machine or resource A server can allocate more than the reservation to a virtual machine but it never allocates more than the limit even if there are unused resources on the system The limit is expressed in concrete units (megahertz or megabytes) The default value for limits is ldquounlimitedrdquo When the memory value is unlimited the amount of memory configured for the virtual machine when it was created becomes its effective limit

Shares Shares specify the relative importance of a virtual machine or resource pool If a virtual machine has twice as many shares of a resource as another virtual machine it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources

Memory Overhead Reservation Memory overhead reservation is the amount of consumed memory being used for virtualization purposes Overhead memory includes space reserved for the virtual machine frame buffer and various virtualization data structures such as shadow page tables

Table 2 vSphere Resource Management Constructs

T E C H N I C A L W H I T E P A P E R 7

VMware vCloud Director Resource Allocation Models

vCloud Director ArchitectureThe vSphere infrastructure provides the foundation of the vCloud Director architecture Building a vCloud service requires an understanding of the individual constructs of the entire architecture

The vSphere layer provides a consumable set of resources available to the VMware vCloud environment and will present these resources to a provider VDC The provider VDC supplies an abstraction layer for resources As each organization VDC withdraws resources from its provider VDC the provider VDC creates the association between the organization VDC and resources at the vSphere layer The vCloud allocation model defined on each organization VDC dictates how the resources of the vSphere layer are consumed and it influences the availability of resources to the other organization VDCs sharing the same provider VDC

O

rgan

izat

ion

Virtual D

atacenter Allocation Model Provider Virtual Datacenter

vSphere R

esou

rces

ConstructInterdynamics

Figure 1 Cycle of vSphere and vCloud Construct Interdynamics

This interaction of vSphere and vCloud constructs determines the flexibility of the resource availability within the vCloud environment And eventually it is the resource availability that drives the consolidation ratio of the organization VDCs within a provider VDC The remaining chapters expand on the architecture of the provider VDC and how vCloud allocation models consume resources

T E C H N I C A L W H I T E P A P E R 8

VMware vCloud Director Resource Allocation Models

Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores

Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering

EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC

Cluster

Storage

Datastores

Provider Virtual Datacenter

Compute

ResourcePool

Figure 2 Provider VDC Resource Mapping

A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster

Provider VDC

Resource Pool

DRS Cluster

Map to

Figure 3 Mapping Provider VDC to Cluster or Resource Pool

By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool

The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources

T E C H N I C A L W H I T E P A P E R 9

VMware vCloud Director Resource Allocation Models

NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible

vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool

Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources

An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations

Organization VDC

Organization Customer 1

GoldProvider

BronzeProvider

Provider VDC

Organization VDC

Organization VDC

Provider VDC

Figure 4 Multiple Organization VDCs

Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models

bullPay-as-you-go

bullAllocation-pool

bullReservation-pool

T E C H N I C A L W H I T E P A P E R 1 0

VMware vCloud Director Resource Allocation Models

Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool

Figure 5 Overview vCloud Director Allocation Model

VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING

Allocation Limit

Guarantee Reservation

vCPU speed Limit on CPU virtual machine level

Table 3 Resource-Setting Taxonomy

The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both

T E C H N I C A L W H I T E P A P E R 1 1

VMware vCloud Director Resource Allocation Models

Resource Pool Resource Pool Resource Pool

VM VM VM VM VM VMVM

Pay-As-You-Go Model

Virtual MachinendashLevelSettings

Virtual Machinendash and Resource PoolndashLevel

Settings

Resource PoolndashLevelSettings

Reservation-Pool Model Allocation-Pool Model

VM VM

Figure 6 Overview of Allocation Models and vSphere Settings

Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo

NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC

Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC

The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Table 4 Provider VDC Resources

T E C H N I C A L W H I T E P A P E R 1 2

VMware vCloud Director Resource Allocation Models

Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults

VM VM VM

MemoryReservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

CPU Limit

CPU Reservation

Resource Pool

Figure 7 Pay-As-You-Go Model Default Settings

Figure 8 Pay-As-You-Go Model Default Settings User Interface

CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit

T E C H N I C A L W H I T E P A P E R 1 3

VMware vCloud Director Resource Allocation Models

EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent

Figure 9 Pay-As-You-Go Virtual Machine CPU Settings

Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface

EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface

Figure 10 Pay-As-You-Go Virtual Machine Memory Settings

T E C H N I C A L W H I T E P A P E R 1 4

VMware vCloud Director Resource Allocation Models

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool NA NA Unlimited NA Enabled

Virtual Machine 026GHz per vCPU

0 NA 100 NA

Table 5 Pay-As-You-Go Default Settings

A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent

Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

Figure 11 Reservation Pool Model Default Settings

T E C H N I C A L W H I T E P A P E R 1 5

VMware vCloud Director Resource Allocation Models

Figure 12 Reservation Pool Model Default Settings User Interface

NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)

The reservation pool model applies resource allocation settings on the resource pool only

CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level

Figure 13 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 6: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 6

VMware vCloud Director Resource Allocation Models

vSphere Resource Constructs Table 2 describes the resource management constructs in vSphere

VSPHERE RESOURCE CONSTRUCT

DESCRIPTION

Distributed Resource Scheduler Cluster

A vSphere DRS cluster is a collection of vSphere hosts with shared resources It must be created first to obtain the benefits of cluster-level resource management such as resource pools

Resource Pools A resource pool is a logical abstraction of physical resources for flexible management of resources Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources

Reservation A reservation specifies the guaranteed minimum physical resource allocation for a virtual machine or resource pool A power-on operation of a virtual machine (via vCenter Server or directly accessing the vSphere host with VMware vSphere Clienttrade) will succeed only if there are enough unreserved resources to satisfy the reservation needed by the virtual machine The server guarantees that amount even when the physical server is heavily loaded The reservation is expressed in physical units (megahertz or megabytes)

Limit Limit specifies an upper boundary for CPU or memory resources that can be allocated to a virtual machine or resource A server can allocate more than the reservation to a virtual machine but it never allocates more than the limit even if there are unused resources on the system The limit is expressed in concrete units (megahertz or megabytes) The default value for limits is ldquounlimitedrdquo When the memory value is unlimited the amount of memory configured for the virtual machine when it was created becomes its effective limit

Shares Shares specify the relative importance of a virtual machine or resource pool If a virtual machine has twice as many shares of a resource as another virtual machine it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources

Memory Overhead Reservation Memory overhead reservation is the amount of consumed memory being used for virtualization purposes Overhead memory includes space reserved for the virtual machine frame buffer and various virtualization data structures such as shadow page tables

Table 2 vSphere Resource Management Constructs

T E C H N I C A L W H I T E P A P E R 7

VMware vCloud Director Resource Allocation Models

vCloud Director ArchitectureThe vSphere infrastructure provides the foundation of the vCloud Director architecture Building a vCloud service requires an understanding of the individual constructs of the entire architecture

The vSphere layer provides a consumable set of resources available to the VMware vCloud environment and will present these resources to a provider VDC The provider VDC supplies an abstraction layer for resources As each organization VDC withdraws resources from its provider VDC the provider VDC creates the association between the organization VDC and resources at the vSphere layer The vCloud allocation model defined on each organization VDC dictates how the resources of the vSphere layer are consumed and it influences the availability of resources to the other organization VDCs sharing the same provider VDC

O

rgan

izat

ion

Virtual D

atacenter Allocation Model Provider Virtual Datacenter

vSphere R

esou

rces

ConstructInterdynamics

Figure 1 Cycle of vSphere and vCloud Construct Interdynamics

This interaction of vSphere and vCloud constructs determines the flexibility of the resource availability within the vCloud environment And eventually it is the resource availability that drives the consolidation ratio of the organization VDCs within a provider VDC The remaining chapters expand on the architecture of the provider VDC and how vCloud allocation models consume resources

T E C H N I C A L W H I T E P A P E R 8

VMware vCloud Director Resource Allocation Models

Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores

Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering

EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC

Cluster

Storage

Datastores

Provider Virtual Datacenter

Compute

ResourcePool

Figure 2 Provider VDC Resource Mapping

A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster

Provider VDC

Resource Pool

DRS Cluster

Map to

Figure 3 Mapping Provider VDC to Cluster or Resource Pool

By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool

The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources

T E C H N I C A L W H I T E P A P E R 9

VMware vCloud Director Resource Allocation Models

NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible

vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool

Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources

An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations

Organization VDC

Organization Customer 1

GoldProvider

BronzeProvider

Provider VDC

Organization VDC

Organization VDC

Provider VDC

Figure 4 Multiple Organization VDCs

Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models

bullPay-as-you-go

bullAllocation-pool

bullReservation-pool

T E C H N I C A L W H I T E P A P E R 1 0

VMware vCloud Director Resource Allocation Models

Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool

Figure 5 Overview vCloud Director Allocation Model

VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING

Allocation Limit

Guarantee Reservation

vCPU speed Limit on CPU virtual machine level

Table 3 Resource-Setting Taxonomy

The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both

T E C H N I C A L W H I T E P A P E R 1 1

VMware vCloud Director Resource Allocation Models

Resource Pool Resource Pool Resource Pool

VM VM VM VM VM VMVM

Pay-As-You-Go Model

Virtual MachinendashLevelSettings

Virtual Machinendash and Resource PoolndashLevel

Settings

Resource PoolndashLevelSettings

Reservation-Pool Model Allocation-Pool Model

VM VM

Figure 6 Overview of Allocation Models and vSphere Settings

Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo

NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC

Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC

The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Table 4 Provider VDC Resources

T E C H N I C A L W H I T E P A P E R 1 2

VMware vCloud Director Resource Allocation Models

Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults

VM VM VM

MemoryReservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

CPU Limit

CPU Reservation

Resource Pool

Figure 7 Pay-As-You-Go Model Default Settings

Figure 8 Pay-As-You-Go Model Default Settings User Interface

CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit

T E C H N I C A L W H I T E P A P E R 1 3

VMware vCloud Director Resource Allocation Models

EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent

Figure 9 Pay-As-You-Go Virtual Machine CPU Settings

Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface

EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface

Figure 10 Pay-As-You-Go Virtual Machine Memory Settings

T E C H N I C A L W H I T E P A P E R 1 4

VMware vCloud Director Resource Allocation Models

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool NA NA Unlimited NA Enabled

Virtual Machine 026GHz per vCPU

0 NA 100 NA

Table 5 Pay-As-You-Go Default Settings

A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent

Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

Figure 11 Reservation Pool Model Default Settings

T E C H N I C A L W H I T E P A P E R 1 5

VMware vCloud Director Resource Allocation Models

Figure 12 Reservation Pool Model Default Settings User Interface

NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)

The reservation pool model applies resource allocation settings on the resource pool only

CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level

Figure 13 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 7: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 7

VMware vCloud Director Resource Allocation Models

vCloud Director ArchitectureThe vSphere infrastructure provides the foundation of the vCloud Director architecture Building a vCloud service requires an understanding of the individual constructs of the entire architecture

The vSphere layer provides a consumable set of resources available to the VMware vCloud environment and will present these resources to a provider VDC The provider VDC supplies an abstraction layer for resources As each organization VDC withdraws resources from its provider VDC the provider VDC creates the association between the organization VDC and resources at the vSphere layer The vCloud allocation model defined on each organization VDC dictates how the resources of the vSphere layer are consumed and it influences the availability of resources to the other organization VDCs sharing the same provider VDC

O

rgan

izat

ion

Virtual D

atacenter Allocation Model Provider Virtual Datacenter

vSphere R

esou

rces

ConstructInterdynamics

Figure 1 Cycle of vSphere and vCloud Construct Interdynamics

This interaction of vSphere and vCloud constructs determines the flexibility of the resource availability within the vCloud environment And eventually it is the resource availability that drives the consolidation ratio of the organization VDCs within a provider VDC The remaining chapters expand on the architecture of the provider VDC and how vCloud allocation models consume resources

T E C H N I C A L W H I T E P A P E R 8

VMware vCloud Director Resource Allocation Models

Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores

Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering

EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC

Cluster

Storage

Datastores

Provider Virtual Datacenter

Compute

ResourcePool

Figure 2 Provider VDC Resource Mapping

A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster

Provider VDC

Resource Pool

DRS Cluster

Map to

Figure 3 Mapping Provider VDC to Cluster or Resource Pool

By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool

The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources

T E C H N I C A L W H I T E P A P E R 9

VMware vCloud Director Resource Allocation Models

NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible

vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool

Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources

An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations

Organization VDC

Organization Customer 1

GoldProvider

BronzeProvider

Provider VDC

Organization VDC

Organization VDC

Provider VDC

Figure 4 Multiple Organization VDCs

Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models

bullPay-as-you-go

bullAllocation-pool

bullReservation-pool

T E C H N I C A L W H I T E P A P E R 1 0

VMware vCloud Director Resource Allocation Models

Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool

Figure 5 Overview vCloud Director Allocation Model

VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING

Allocation Limit

Guarantee Reservation

vCPU speed Limit on CPU virtual machine level

Table 3 Resource-Setting Taxonomy

The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both

T E C H N I C A L W H I T E P A P E R 1 1

VMware vCloud Director Resource Allocation Models

Resource Pool Resource Pool Resource Pool

VM VM VM VM VM VMVM

Pay-As-You-Go Model

Virtual MachinendashLevelSettings

Virtual Machinendash and Resource PoolndashLevel

Settings

Resource PoolndashLevelSettings

Reservation-Pool Model Allocation-Pool Model

VM VM

Figure 6 Overview of Allocation Models and vSphere Settings

Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo

NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC

Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC

The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Table 4 Provider VDC Resources

T E C H N I C A L W H I T E P A P E R 1 2

VMware vCloud Director Resource Allocation Models

Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults

VM VM VM

MemoryReservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

CPU Limit

CPU Reservation

Resource Pool

Figure 7 Pay-As-You-Go Model Default Settings

Figure 8 Pay-As-You-Go Model Default Settings User Interface

CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit

T E C H N I C A L W H I T E P A P E R 1 3

VMware vCloud Director Resource Allocation Models

EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent

Figure 9 Pay-As-You-Go Virtual Machine CPU Settings

Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface

EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface

Figure 10 Pay-As-You-Go Virtual Machine Memory Settings

T E C H N I C A L W H I T E P A P E R 1 4

VMware vCloud Director Resource Allocation Models

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool NA NA Unlimited NA Enabled

Virtual Machine 026GHz per vCPU

0 NA 100 NA

Table 5 Pay-As-You-Go Default Settings

A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent

Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

Figure 11 Reservation Pool Model Default Settings

T E C H N I C A L W H I T E P A P E R 1 5

VMware vCloud Director Resource Allocation Models

Figure 12 Reservation Pool Model Default Settings User Interface

NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)

The reservation pool model applies resource allocation settings on the resource pool only

CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level

Figure 13 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 8: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 8

VMware vCloud Director Resource Allocation Models

Architecting the Provider Virtual DatacenterA provider VDC is a collection and an abstraction of vSphere resources Virtual datacenters deal with two types of resources compute and storage vSphere DRS clusters or resource pools provide the CPU and memory resources while storage resources are provided by datastores

Abstracting these infrastructure resources into a single management construct enables the pooling of compute and storage resources of similar performance or availability levels that can be mapped to a specific service offering

EXAMPLEDifferentiationbyperformancemightbeaFlash-based(SSD)storagepoolforagoldproviderVDCasopposedtoaSATA-basedstoragepoolforabronzeproviderVDC

Cluster

Storage

Datastores

Provider Virtual Datacenter

Compute

ResourcePool

Figure 2 Provider VDC Resource Mapping

A provider VDC can be mapped directly to a vSphere DRS cluster or to a resource pool within a vSphere DRS cluster

Provider VDC

Resource Pool

DRS Cluster

Map to

Figure 3 Mapping Provider VDC to Cluster or Resource Pool

By mapping the provider VDC to a vSphere DRS cluster all compute and storage resources are dedicated to the provider VDC This enables efficient distribution and allocation of resources to various organizations If additional resources are required hosts and storage can be added to the vSphere DRS cluster New physical resources added to the cluster are automatically integrated to the provider VDC When the provider VDC is mapped to resource pool level additional physical resources must be configured manually by updating the reservation and limits of the resource pool

The partitioning of cluster resources by mapping provider VDCs to resource pools increases the complexity of managing compute and storage resources

T E C H N I C A L W H I T E P A P E R 9

VMware vCloud Director Resource Allocation Models

NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible

vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool

Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources

An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations

Organization VDC

Organization Customer 1

GoldProvider

BronzeProvider

Provider VDC

Organization VDC

Organization VDC

Provider VDC

Figure 4 Multiple Organization VDCs

Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models

bullPay-as-you-go

bullAllocation-pool

bullReservation-pool

T E C H N I C A L W H I T E P A P E R 1 0

VMware vCloud Director Resource Allocation Models

Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool

Figure 5 Overview vCloud Director Allocation Model

VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING

Allocation Limit

Guarantee Reservation

vCPU speed Limit on CPU virtual machine level

Table 3 Resource-Setting Taxonomy

The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both

T E C H N I C A L W H I T E P A P E R 1 1

VMware vCloud Director Resource Allocation Models

Resource Pool Resource Pool Resource Pool

VM VM VM VM VM VMVM

Pay-As-You-Go Model

Virtual MachinendashLevelSettings

Virtual Machinendash and Resource PoolndashLevel

Settings

Resource PoolndashLevelSettings

Reservation-Pool Model Allocation-Pool Model

VM VM

Figure 6 Overview of Allocation Models and vSphere Settings

Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo

NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC

Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC

The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Table 4 Provider VDC Resources

T E C H N I C A L W H I T E P A P E R 1 2

VMware vCloud Director Resource Allocation Models

Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults

VM VM VM

MemoryReservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

CPU Limit

CPU Reservation

Resource Pool

Figure 7 Pay-As-You-Go Model Default Settings

Figure 8 Pay-As-You-Go Model Default Settings User Interface

CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit

T E C H N I C A L W H I T E P A P E R 1 3

VMware vCloud Director Resource Allocation Models

EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent

Figure 9 Pay-As-You-Go Virtual Machine CPU Settings

Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface

EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface

Figure 10 Pay-As-You-Go Virtual Machine Memory Settings

T E C H N I C A L W H I T E P A P E R 1 4

VMware vCloud Director Resource Allocation Models

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool NA NA Unlimited NA Enabled

Virtual Machine 026GHz per vCPU

0 NA 100 NA

Table 5 Pay-As-You-Go Default Settings

A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent

Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

Figure 11 Reservation Pool Model Default Settings

T E C H N I C A L W H I T E P A P E R 1 5

VMware vCloud Director Resource Allocation Models

Figure 12 Reservation Pool Model Default Settings User Interface

NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)

The reservation pool model applies resource allocation settings on the resource pool only

CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level

Figure 13 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 9: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 9

VMware vCloud Director Resource Allocation Models

NOTEItisavSpherebestpracticetoconnectdatastorestoallhostsintheclusterenablingvSphereDRStodistributevirtualmachinesasoptimallyaspossible

vCloud Director utilizes vSphere resource pools to partition the resources of the provider VDC for each organization If the provider VDC is mapped to a resource pool only the reserved resources assigned to the parent resource pool is used and distributed among the organization VDCs The resource pools belonging to different organization VDCs become children of the parent provider VDC resource pool

Organization Virtual DatacentersOrganization VDCs are used by vCloud Director to partition provider VDCs As mentioned previously vCloud Director uses vSphere resource pools as the basic construct to partition these resources

An organization can have multiple organization VDCs each utilizing resources from a different provider VDC offering different classes of service based on associated performance availability and cost characteristics A provider VDC can present its resources to multiple organization VDCs belonging to different organizations

Organization VDC

Organization Customer 1

GoldProvider

BronzeProvider

Provider VDC

Organization VDC

Organization VDC

Provider VDC

Figure 4 Multiple Organization VDCs

Each organization VDC is configured based on an allocation model An allocation model defines how resources are allocated from the provider VDC An organization VDC must be configured with one of the following three allocation models

bullPay-as-you-go

bullAllocation-pool

bullReservation-pool

T E C H N I C A L W H I T E P A P E R 1 0

VMware vCloud Director Resource Allocation Models

Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool

Figure 5 Overview vCloud Director Allocation Model

VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING

Allocation Limit

Guarantee Reservation

vCPU speed Limit on CPU virtual machine level

Table 3 Resource-Setting Taxonomy

The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both

T E C H N I C A L W H I T E P A P E R 1 1

VMware vCloud Director Resource Allocation Models

Resource Pool Resource Pool Resource Pool

VM VM VM VM VM VMVM

Pay-As-You-Go Model

Virtual MachinendashLevelSettings

Virtual Machinendash and Resource PoolndashLevel

Settings

Resource PoolndashLevelSettings

Reservation-Pool Model Allocation-Pool Model

VM VM

Figure 6 Overview of Allocation Models and vSphere Settings

Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo

NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC

Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC

The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Table 4 Provider VDC Resources

T E C H N I C A L W H I T E P A P E R 1 2

VMware vCloud Director Resource Allocation Models

Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults

VM VM VM

MemoryReservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

CPU Limit

CPU Reservation

Resource Pool

Figure 7 Pay-As-You-Go Model Default Settings

Figure 8 Pay-As-You-Go Model Default Settings User Interface

CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit

T E C H N I C A L W H I T E P A P E R 1 3

VMware vCloud Director Resource Allocation Models

EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent

Figure 9 Pay-As-You-Go Virtual Machine CPU Settings

Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface

EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface

Figure 10 Pay-As-You-Go Virtual Machine Memory Settings

T E C H N I C A L W H I T E P A P E R 1 4

VMware vCloud Director Resource Allocation Models

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool NA NA Unlimited NA Enabled

Virtual Machine 026GHz per vCPU

0 NA 100 NA

Table 5 Pay-As-You-Go Default Settings

A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent

Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

Figure 11 Reservation Pool Model Default Settings

T E C H N I C A L W H I T E P A P E R 1 5

VMware vCloud Director Resource Allocation Models

Figure 12 Reservation Pool Model Default Settings User Interface

NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)

The reservation pool model applies resource allocation settings on the resource pool only

CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level

Figure 13 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 10: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 1 0

VMware vCloud Director Resource Allocation Models

Allocation ModelsOrganizations can consume resources from multiple organization VDCs using one of three resource allocation models pay-as-you-go allocation-pool and reservation-pool

Figure 5 Overview vCloud Director Allocation Model

VCLOUD DIRECTOR RESOURCE SETTING VSPHERE RESOURCE SETTING

Allocation Limit

Guarantee Reservation

vCPU speed Limit on CPU virtual machine level

Table 3 Resource-Setting Taxonomy

The distinction between each of the allocation models is in how resources are consumed vCloud Director enables each allocation model to configure a guaranteed amount (reservation) and a maximum amount (limit) of physical resources that are available to the organization VDC To guarantee physical resources vSphere can set a reservation on resource pool level and virtual machine level A limit is set on resource pool level and virtual machine level if a maximum amount of resources is defined in the organization VDC Depending on the chosen allocation model reservations and limits will be set on resource pool level virtual machine level or both

T E C H N I C A L W H I T E P A P E R 1 1

VMware vCloud Director Resource Allocation Models

Resource Pool Resource Pool Resource Pool

VM VM VM VM VM VMVM

Pay-As-You-Go Model

Virtual MachinendashLevelSettings

Virtual Machinendash and Resource PoolndashLevel

Settings

Resource PoolndashLevelSettings

Reservation-Pool Model Allocation-Pool Model

VM VM

Figure 6 Overview of Allocation Models and vSphere Settings

Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo

NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC

Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC

The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Table 4 Provider VDC Resources

T E C H N I C A L W H I T E P A P E R 1 2

VMware vCloud Director Resource Allocation Models

Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults

VM VM VM

MemoryReservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

CPU Limit

CPU Reservation

Resource Pool

Figure 7 Pay-As-You-Go Model Default Settings

Figure 8 Pay-As-You-Go Model Default Settings User Interface

CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit

T E C H N I C A L W H I T E P A P E R 1 3

VMware vCloud Director Resource Allocation Models

EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent

Figure 9 Pay-As-You-Go Virtual Machine CPU Settings

Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface

EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface

Figure 10 Pay-As-You-Go Virtual Machine Memory Settings

T E C H N I C A L W H I T E P A P E R 1 4

VMware vCloud Director Resource Allocation Models

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool NA NA Unlimited NA Enabled

Virtual Machine 026GHz per vCPU

0 NA 100 NA

Table 5 Pay-As-You-Go Default Settings

A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent

Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

Figure 11 Reservation Pool Model Default Settings

T E C H N I C A L W H I T E P A P E R 1 5

VMware vCloud Director Resource Allocation Models

Figure 12 Reservation Pool Model Default Settings User Interface

NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)

The reservation pool model applies resource allocation settings on the resource pool only

CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level

Figure 13 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 11: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 1 1

VMware vCloud Director Resource Allocation Models

Resource Pool Resource Pool Resource Pool

VM VM VM VM VM VMVM

Pay-As-You-Go Model

Virtual MachinendashLevelSettings

Virtual Machinendash and Resource PoolndashLevel

Settings

Resource PoolndashLevelSettings

Reservation-Pool Model Allocation-Pool Model

VM VM

Figure 6 Overview of Allocation Models and vSphere Settings

Each allocation model enables specifying allocation and guarantees on CPU and memory resources vCloud Director uses the terminology ldquoallocationsrdquo and ldquoguaranteesrdquo whereas the vSphere layer provides ldquolimitsrdquo and ldquoreservationsrdquo

NOTEByleveragingresourceallocationsettingsatthevSpherelayeronlytheallocationofphysicalresourcesisdictatedAcquisitionofvirtualresourcessuchasmemoryreclamationtechniquesofballooningorswappingatthevSpherehostlevelisstillpossibledependingonthechosenresourcesettingsOversubscriptiononaresourcepoollevelorwithintheresourcepoolcanimpactotherlayersofthevCloudinfrastructureandmightimpactothervirtualmachinesoutsidetheorganizationVDC

Each allocation model contains the setting maximum number of virtual machines This setting limits the number of virtual machines that can be deployed within the organization VDC

The provider VDC configuration in the vCloud environment used to provide the screenshots and used in the examples in the following sections contains the following resources

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Table 4 Provider VDC Resources

T E C H N I C A L W H I T E P A P E R 1 2

VMware vCloud Director Resource Allocation Models

Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults

VM VM VM

MemoryReservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

CPU Limit

CPU Reservation

Resource Pool

Figure 7 Pay-As-You-Go Model Default Settings

Figure 8 Pay-As-You-Go Model Default Settings User Interface

CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit

T E C H N I C A L W H I T E P A P E R 1 3

VMware vCloud Director Resource Allocation Models

EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent

Figure 9 Pay-As-You-Go Virtual Machine CPU Settings

Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface

EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface

Figure 10 Pay-As-You-Go Virtual Machine Memory Settings

T E C H N I C A L W H I T E P A P E R 1 4

VMware vCloud Director Resource Allocation Models

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool NA NA Unlimited NA Enabled

Virtual Machine 026GHz per vCPU

0 NA 100 NA

Table 5 Pay-As-You-Go Default Settings

A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent

Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

Figure 11 Reservation Pool Model Default Settings

T E C H N I C A L W H I T E P A P E R 1 5

VMware vCloud Director Resource Allocation Models

Figure 12 Reservation Pool Model Default Settings User Interface

NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)

The reservation pool model applies resource allocation settings on the resource pool only

CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level

Figure 13 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 12: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 1 2

VMware vCloud Director Resource Allocation Models

Pay-As-You-Go Model SettingsThe pay-as-you-go model provides the illusion of an unlimited resource pool This model maps to a resource pool with no configured reservations or limits Resources are committed only when vApps are deployed in the organization VDC The pay-as-you-go model applies per virtual machine resource allocation settings only It creates a resource pool and keeps all settings of that pool as defaults

VM VM VM

MemoryReservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

CPU Limit

CPU Reservation

Resource Pool

Figure 7 Pay-As-You-Go Model Default Settings

Figure 8 Pay-As-You-Go Model Default Settings User Interface

CPU resources The pay-as-you-go model sets no CPU limit on resource pool level but it enables configuring the vCPU limit via the vCPU speed setting on virtual machine level By default a 026GHz-per-vCPU recommendation is made By default the guarantee is set to 0 percent resulting in no CPU reservation on virtual machine level The allocation of protected physical resources is capped by limit

T E C H N I C A L W H I T E P A P E R 1 3

VMware vCloud Director Resource Allocation Models

EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent

Figure 9 Pay-As-You-Go Virtual Machine CPU Settings

Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface

EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface

Figure 10 Pay-As-You-Go Virtual Machine Memory Settings

T E C H N I C A L W H I T E P A P E R 1 4

VMware vCloud Director Resource Allocation Models

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool NA NA Unlimited NA Enabled

Virtual Machine 026GHz per vCPU

0 NA 100 NA

Table 5 Pay-As-You-Go Default Settings

A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent

Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

Figure 11 Reservation Pool Model Default Settings

T E C H N I C A L W H I T E P A P E R 1 5

VMware vCloud Director Resource Allocation Models

Figure 12 Reservation Pool Model Default Settings User Interface

NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)

The reservation pool model applies resource allocation settings on the resource pool only

CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level

Figure 13 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 13: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 1 3

VMware vCloud Director Resource Allocation Models

EXAMPLEIfavirtualmachineisconfiguredwithtwovCPUsthelimitonthevirtualmachineissetto520MHzForadual-vCPUvirtualmachine530MHzofprotectedphysicalCPUresourcesareavailableifCPUresourcesguaranteedissetto100percent

Figure 9 Pay-As-You-Go Virtual Machine CPU Settings

Memory resources The Unlimited setting is enabled on resource pool level Depending on the activity of its virtual machines Unlimited enables the resource pool to allocate all the unreserved resources within the cluster On virtual machine level a limit is set identical to the configured memory size of the virtual machine This setting is not adjustable via the vCloud Director user interface

EXAMPLEA4GBlimitissetonavirtualmachineconfiguredwith4GBThepay-as-you-gomodelenablesconfiguringonlyvirtualmachinendashlevelsettingsandwillnotsetareservationonresourcepoollevelTheuserinterfaceenablesconfigurationofamemoryreservationrecommendedat100percentonvirtualmachinelevelThisdefaultsettingresultsinafullmemoryreservationonthevirtualmachinelevelInthisexamplea4GBlimitissetonavirtualmachineconfiguredwith4GBThissettingcannotbechangedviathevCloudDirectoruserinterface

Figure 10 Pay-As-You-Go Virtual Machine Memory Settings

T E C H N I C A L W H I T E P A P E R 1 4

VMware vCloud Director Resource Allocation Models

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool NA NA Unlimited NA Enabled

Virtual Machine 026GHz per vCPU

0 NA 100 NA

Table 5 Pay-As-You-Go Default Settings

A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent

Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

Figure 11 Reservation Pool Model Default Settings

T E C H N I C A L W H I T E P A P E R 1 5

VMware vCloud Director Resource Allocation Models

Figure 12 Reservation Pool Model Default Settings User Interface

NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)

The reservation pool model applies resource allocation settings on the resource pool only

CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level

Figure 13 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 14: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 1 4

VMware vCloud Director Resource Allocation Models

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool NA NA Unlimited NA Enabled

Virtual Machine 026GHz per vCPU

0 NA 100 NA

Table 5 Pay-As-You-Go Default Settings

A limit is configured equal to the configured size of the virtual machine This user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is enabled on both CPU and memory resources of the resource pool This enables the resource pool to allocate required unreserved resources from its parent

Reservation Pool Model SettingsReservation pool resources allocated to the organization VDC are completely dedicated This is identical to an allocation pool with all guarantees set to 100 percent Reservation pool VDCs map to resource pools with the reservations set equivalent to the limits

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

Figure 11 Reservation Pool Model Default Settings

T E C H N I C A L W H I T E P A P E R 1 5

VMware vCloud Director Resource Allocation Models

Figure 12 Reservation Pool Model Default Settings User Interface

NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)

The reservation pool model applies resource allocation settings on the resource pool only

CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level

Figure 13 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 15: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 1 5

VMware vCloud Director Resource Allocation Models

Figure 12 Reservation Pool Model Default Settings User Interface

NOTEThevCloudDirectorreservationpoolmodeluserinterfaceusesthetermldquoallocationrdquotoidentifyaguarantee(reservation)

The reservation pool model applies resource allocation settings on the resource pool only

CPU resources The reservation pool provides the recommendation of setting the CPU reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the reservation No reservation and limits are set on the virtual machine level

Figure 13 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhensettingtheCPUallocationof3GHzaCPUreservationandlimitof3GHzissetontheresourcepoollevel

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 16: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 1 6

VMware vCloud Director Resource Allocation Models

Memory resources Similar to CPU settings the reservation pool provides the recommendation of setting the memory reservation equal to 20 percent of the available resources from the provider VDC the limit is set equal to the configured limit No reservation and limits are set on the virtual machine level

Figure 14 Resource PoolndashLevel Memory Resource Settings

EXAMPLEWhensettingthememoryallocationof10GBaCPUreservationandlimitof10GBissetontheresourcepoollevel

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 20 of PVDC NA NA 20 of PVDC Disabled

Virtual Machine NA NA NA NA NA

Table 6 Reservation Pool Default Settings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationBy default expandable reservation is disabled on both CPU and memory resources of the resource pool The vCloud Director user interface does not offer the possibility to adjust this setting

Custom PerndashVirtual Machine Resource SettingsBy default the reservation pool allocation model does not apply perndashvirtual machine resource allocation settings but vCloud Director enables custom perndashvirtual machine resource settings vCloud Director does not allow settings to be changed on a per-vApp basis If a vApp contains multiple virtual machines each virtual machine must be edited separately

Because custom resource settings are allowed at the perndashvirtual machine level care must be taken when importing templates If the imported template has preconfigured perndashvirtual machine resource settings these settings will be maintained All subsequent vApps deployed from this template will be configured with the custom resource settings

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 17: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 1 7

VMware vCloud Director Resource Allocation Models

NOTEBothpay-as-you-goandallocationpoolmodelsdonotapplythetemplatesettingstonewvirtualmachinesEachmodelreconfiguresthevirtualmachineandappliestheorganizationVDCresourcesettingsduringanypower-onoperation

To adjust the resource allocation settings go to the My cloud view This shows all virtual machines registered to the organization VDC Select the virtual machine right-click it select properties and go to the Resource Allocation tab

Figure 15 Virtual Machine Custom Resource Allocation Tab

Allocation Pool Model SettingsAn allocation pool is a pool of allocated resources with a certain percentage guaranteed The percentage guaranteed directly translates into reservations configured on the child resource pool The difference between the reservation and the limit is in the resources that can be oversubscribed

Figure 16 Allocation Pool Default Settings User Interface

The allocation pool model applies resource allocation settings on resource pool and virtual machine levels

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 18: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 1 8

VMware vCloud Director Resource Allocation Models

MemoryReservation

MemoryLimit

Resource PoolndashLevel Settings

Virtual MachinendashLevel Settings

Resource Pool

VM VM VM

MemoryReservation

MemoryLimit

CPU Limit

CPU Reservation

Figure 17 Allocation Pool Model Default Settings

CPU resources The allocation pool model provides a recommendation to set the CPU limit on the resource pool level equal to 10 percent of the available resources from the provider VDC No limit is set on the virtual machine level this setting cannot be changed via the vCloud Director user interface The allocation pool model provides the option to set a CPU reservation but recommends a 0 percent CPU reservation setting The setting applies a CPU reservation on the resource pool level but not on the virtual machine level The user interface does not provide options to change the virtual machine CPU reservation setting

Figure 18 Resource PoolndashLevel CPU Resource Settings

EXAMPLEWhentheCPUallocationissetto319GHzanda0percentguaranteeisspecifiedtheresourcepoolisconfiguredwithaCPUlimitof319GHzNoreservationissetattheresourcepoollevelTheunlimitedsettingisenabledatthevirtualmachinelevelduetothe0percentguaranteedsettingattheallocationpoollevelNewvirtualmachinesareconfiguredwithoutaCPUreservation

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 19: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 1 9

VMware vCloud Director Resource Allocation Models

Figure 19 Virtual MachinendashLevel CPU Resource Settings

Memory resources The allocation pool model provides a recommendation to set the CPU reservation on the resource pool level equal to 20 percent of the available resources from the provider VDC A limit is set identical to the configured memory size of the virtual machine This setting cannot be changed via the vCloud Director user interface By default the allocation pool model provides the recommendation to set the reservation to 100 percent resulting in a memory reservation equal to the configured limit The percentage specified for memory reservation is applied on the virtual machine level as well

Figure 20 Resource PoolndashLevel Memory Resource Settings

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 20: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 2 0

VMware vCloud Director Resource Allocation Models

Figure 21 Virtual MachinendashLevel Memory Resource Settings

EXAMPLEWhenthememoryallocationissetto953GBanda100percentguaranteeisspecifiedtheresourcepoolisconfiguredwithamemorylimitof953GBThememoryreservationissetequaltothelimitInthisexamplethememoryreservationissetto953GBattheresourcepoollevelBothamemorylimitandamemoryreservationaresetoneachnewvirtualmachineThelimitissetequaltotheconfiguredsizeofthevirtualmachineForexamplea4GBvirtualmachineisconfiguredwitha4GBlimitDuetothe100percentmemoryreservationatallocationpoolmodellevelthevirtualmachineisconfiguredwithareservationthatisequaltothelimit4GBinthiscase

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

EXPANDABLE RESERVATION

Resource Pool 10 of PVDC 0 20 of PVDC 100 Disabled

Virtual Machine Unlimited NA 100 100 NA

Table 7 vCPU Resource Mappings

The vCloud Director user interface does not offer the possibility to adjust this setting

Expandable ReservationExpandable reservation is disabled on both CPU and memory resources of the resource pool

Review of Resource DistributionThis section reviews the resource distribution concepts of all the vCloud Director allocation models The primary focus of these examples is on the distribution and consumption of memory resources The reason for the focus on memory is that when resource demand exceeds resource supply the overhead and additional load on the supporting infrastructure is much higher than the overhead when CPU resource demand exceeds CPU resource supply If CPU demand exceeds supply CPU instructions are queued If memory demand exceeds supply multiple memory reclamation techniques are leveraged potentially increasing load on the CPU of the VMware vSphere ESXitrade host and possibly increasing storage load as a result of moving memory into swap files

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 21: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 2 1

VMware vCloud Director Resource Allocation Models

TERM DESCRIPTION

Opportunistic access Resource allocation based on priority and levels of contention Resources are allocated based on the relative share value

Contention Resource demand exceeds resource supply

Table 8 Resource Management Taxonomy

Virtual Machine Distribution on Storage Level vCloud Director distributes virtual machines within the provider VDC based on lowest utilization of available datastores without discriminating as to organization VDC This possibly places virtual machines of two different organizations on a single datastore Resource contention within the organization can impact the performance of virtual machines belonging to other organization VDCs within the same provider VDC

Pay-As-You-Go ModelThe pay-as-you-go model configures resource allocation settings at the virtual machine level only Although a resource pool is created vCloud Director does not alter its default settings This model enables the allocation of available unreserved resources within the provider VDC

The amount of resources allocated by the organization VDC configured with a pay-as-you-go allocation model is determined by the following factors

bullThenumberofvirtualmachinesrunningwithintheresourcepool

bullTheresourceconfigurationofeachvirtualmachinesuchasnumberofvCPUs

bullVirtualmachinememorysizeandspecifiedallocationsettings

Organization VDC

VM VM VM

Provider VDC

VirtualMachineReservedMemory Allocated

Space byOrganization

VDC

UnallocatedldquoGrowthrdquo

Space

UnreservedMemory

Withdrawnfrom Provider

VDC

Figure 22 Resource Distribution of Pay-As-You-Go Model

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 22: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 2 2

VMware vCloud Director Resource Allocation Models

Opportunistic allocation The pay-as-you-go model enables configuring the CPU reservation on virtual machine level only and sets it by default to 0 percent Using the default results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the pay-as-you-go model sets a 100 percent memory reservation If this percentage is reduced the allocation of unreserved resources is based on relative share value during contention of resources The share value on virtual machines is set to normal by default and cannot be changed Therefore access to unreserved resources depends on the activity of the virtual machine its configured size the activity of other virtual machines and resource pools and the availability of resources

The expandable reservation is used to withdraw resources from the provider VDC to satisfy the virtual machine reservation If the provider VDC hosts multiple organization VDCs and expandable reservation is enabled on multiple organization VCDs opportunistic access is present at the resource pool level

Although the pay-as-you-go model uses a resource pool the virtual machinendashlevel reservations ldquoflowrdquo directly through the resource pool to the provider VDC This reduces the amount of resources available to other virtual machines and other organization VDCs running in the provider VDC Physical resources allocated by the virtual machine reservation are available only for that virtual machine and will not be shared with other virtual machines within the organization VDC

VM

Virtual MachinendashLevel Reservation

ResourcesLeft for Other

Virtual Machines or Resource Pools

Provider VDC

Organization VDC

Figure 23 Virtual MachinendashLevel Reservation Extraction

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 23: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 2 3

VMware vCloud Director Resource Allocation Models

Reservation Pool Model The reservation pool model automatically sets the reservation identical to the specified limit resulting in a configuration that does not provide the ability to burst for resources on the provider VDC layer

Organization VDC

VM VM VM

Provider VDC

Distributed to VirtualMachineBased onActivity

LimitedAllocation

Unavailableto Organization

VDC

UnreservedMemory

Withdrawnfrom

ProviderVDC

Figure 24 Resource Pool Model Resource Distribution

The reservation pool model does not configure any virtual machinendashlevel allocation settings This enables a potential oversubscription of resources within the resource pool of the organization VDC Resources assigned to resource pool level float freely within the pool itself to other virtual machines based on demand and priority

The Maximum number of virtual machines setting limits the number of virtual machines that can be deployed within the organization VDC This setting caps only the number of virtual machines within the resource pool It does not limit the resource allocation of each virtual machine This can lead to resource contention which in turn might lead to memory reclamation done by virtual machine swapping Virtual machine swap files are stored within the virtual machine directory Because swap activity generates disk IO utilization this can lead to datastore performance degradation

Allocation Pool Model The allocation pool model user interface enables the configuration of the percentage of guaranteed CPU resources although it recommends a 0 percent default setting This results in opportunistic allocation of CPU resources by the resource pool and virtual machines belonging to the organization VDC

By default the allocation pool model sets a 100 percent memory reservation on both resource pool level and virtual machine level By lowering the default guarantee it enables opportunistic memory allocation on both resource pool level and virtual machine level Creating this burstable space (resources available for opportunistic access) usually provides a higher consolidation ratio of virtual machines However due to the simultaneous configuration of reservation on both resource pool and virtual machine level this is not the case

During a power-on operation admission control checks whether the resource pool can satisfy the virtual machinendashlevel reservation Because expandable reservation is disabled in this model the resource pool is not able to allocate any additional resources from the provider VDC Therefore the virtual machine memory reservation can be satisfied only by the resource poolndashlevel reservation of the organization VDC itself When a virtual machine is using memory protected by a virtual machinendashlevel reservation this memory is withdrawn from the resource poolndashlevel reservation If the resource pool does not have enough available memory to guarantee the virtual machine reservation the power-on operation fails

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 24: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 2 4

VMware vCloud Director Resource Allocation Models

An organization VCD is created with the allocation pool model and the memory allocation is set to 20GB the memory guarantee is set to 50 percent These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB When powering up a 2GB virtual machine 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool

Organization VDC20GB Allocation50 Guarantee

VM12GB

UnreservedMemory

10GB BurstSpace Available

10GBReservedMemoryAvailable

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 25 Admission Control Phase 1

Admission control enables the power-on of virtual machines until the reserved memory pool is reduced to zero Following the previous power-on example virtual machine 2 is powered on The resource pool providing resources to the organization VDC has 9GB available in its pool of reserved memory Admission control enables the power-on operation of the virtual machine because this pool can provide the reserved resources specified by the virtual machinendashlevel reservation

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 25: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 2 5

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM22GB

VM1Unreserved

Memory

10GB BurstSpace Available

9GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

Figure 26 Admission Control Phase 2

During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization VDC This results in admission controlrsquos enabling the power-on of 10 virtual machines

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 26: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 2 6

VMware vCloud Director Resource Allocation Models

Organization VDC20GB Allocation50 Guarantee

VM112GB

UnreservedMemory

10GB BurstSpace Available

0GBReservedMemoryAvailable

Withdrawnfrom

Resource PoolReservation

1GBVirtual MachinendashLevel

Reservation

Admission Control

1GBOpportunistic

Access

VM1

VM2

VM3

VM4

VM5

VM6

VM7

VM8

VM9

VM10

Figure 27 Admission Control Phase 3 ndash Failure

When attempting to deploy virtual machine 11 admission control fails the power-on operation because the organization VDC has no available reserved memory to satisfy the virtual machinendashlevel reservation

NOTEThisscenarioexcludestheimpactofmemoryoverheadreservationofeachvirtualmachineUndernormalcircumstancesthenumberofvirtualmachinesthatcanbepoweredoniscloseto8insteadof10becausethereservedpoolavailabletotheorganizationVDCisusedtosatisfythememoryoverheadreservationofeachvirtualmachineaswell

Because the guarantee setting of the allocation pool model configures resource pool and virtual machine memory reservation settings simultaneously the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting Therefore offering opportunistic access to resources within the organization VDC does not enable an increase in the number of virtual machines within the organization VDC

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 27: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 2 7

VMware vCloud Director Resource Allocation Models

Decreasing the percentage of guaranteed resources provides burstable space that enables increasing the number of organization VDCs within the provider VDC

40GB Allocated100 Guaranteed

ReservedMemory

UnreservedMemory

40GB Allocated100 Guaranteed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

40GB A

lloca

ted

50

Gua

rant

eed

Opp

ortu

nist

ic

Acces

s

Provider VDC80GB

Provider VDC80GB

Organization VDC

Organization VDC

OrgVDC

OrgVDC

OrgVDC

Figure 28 TBD

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 28: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 2 8

VMware vCloud Director Resource Allocation Models

vCloud Allocation Models in PracticeConsumer Functional RequirementsThroughout this paper vCloud allocation models are examined from the perspective of the service provider To understand these in practice it is important to recognize the use cases for each model However instead of defining traditional use cases it is better to view them in terms of the consumerrsquos potential functional requirements to determine the best-fit model

Pay-As-You-Go ModelThe following are among the functional requirements that can drive a pay-as-you-go model

bullCatalogtemplatestorage

bullTemporarytransientneeds

bullLow-costlow-performanceworkloads

Each of the above still can have different perndashvirtual machine settings based on the additional need for performance when aligned with the requirements In some cases this model has no ceiling for the number of virtual machines Depending on the settings it might be the best-performing model on a perndashvirtual machine basis However it might also be the most expensive for the provider to maintain physical resources to satisfy the perndashvirtual machine guarantees

Reservation Pool ModelThe following are among the functional requirements that can drive a reservation pool model

bullProductionworkloadswhereperndashvirtualmachinelevelcontrolisneededsuchasJavaapplications

bullldquoDedicatedrdquoresources

bullTheneedforeasilybudgetedbilling

These examples all share the characteristic of providing 100 percent of the resources up front for the consumer This also enables the consumer themself to take control of their overcommitment or full commitment of physical resources The caution to the provider is the possible overcommitment of their other resources if consumers do not set their perndashvirtual machine options

Allocation Pool ModelA functional requirement that can drive an allocation pool model is the customer requirement to have a certain amount of guaranteed resources available all the time plus having an additional burst space for the virtual machine for occasional workloads such as the following

bullPer-projecthosting

bullSeasonalapplicationsend-of-weekend-of-monthfinancialsystems

bullMinimumguaranteesofperformance

Each of these indicates a consumer need to have a minimum level of performance and consolidation but also to allow for the use of opportunistic space in their VDC

Each model based on the functional requirements in practice can be tailored based on the additional need for total number of virtual machines coupled with performance needs All of these must be factored in when making the decision regarding which model best suits a consumer

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 29: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 2 9

VMware vCloud Director Resource Allocation Models

Catalog Organization Virtual DatacenterAs with any vApp catalogs are stored in an organization VDC To avoid wasting resources it is recommended that the pay-as-you-go model be applied to the organization VDC hosting shared master catalog virtual machines This model does not set any memory reservation on resource pool level as opposed to the reservation pool and allocation pool models These other models set memory reservations on resource pool levels and are withdrawn from the provider VDC regardless of active workload in the resource pool The pay-as-you-go model sets a memory reservation equal to the configured size of the virtual machine However because the template is not powered on the reservation is inactive and will not extract available resources from the provider VDC

vCloud System Virtual Datacenter Every vCloud installation includes a system VDC resource pool which is used exclusively to store all organization network VMware vShield Edgetrade devices The resource pool is set to expandable reservation with no limits on CPU and memory A vShield Edge device is a preconfigured virtual appliance that provides secure multitenancy through the use of network address translation and firewall rules

RESOURCE CONFIGURATION

CPU 1

Memory 256MB

Table 9 vShield Virtual Hardware Configuration

In addition there are the following perndashvirtual machine reservations and limits set on these that cannot be edited

SETTING LIMIT RESERVATION

CPU Unlimited 64MHz

Memory 256MB 64MB

Table 10 vShield Edge Resource Allocation Settings

The system VDC must provide at least 90MB of memory to power on to each vShield Edge device based on this configuration Each vShield Edge device must be provided with at least 25MB of physical memory to satisfy its virtual machine overhead memory reservation

A system VDC is a child resource pool of the provider VDC and is placed next to other organization VDC resource pools The system VDC withdraws resources from the provider VDC rendering these resources unavailable to other organization VDCs Although the vShield Edge device configuration seems relatively small having many organizations with organization networks in a single provider VDC can impact overall resource capacity

EXAMPLEInamediumtolargeproviderwithhundredsoforganizationsonasingleclusterthesenumberscanbeimpactfulasillustratedinthefollowing

250 organizations with one NAT network each =

bull250vShieldEdgedevicesdeployed

bullApproximately21GBofreservedmemoryneeded(includingoverheadreservation)

bullApproximately16GHzofreservedCPUresources

NOTETheprecedingvShieldEdgeconfigurationalsowillapplytoanyvAppnetworks

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 30: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 3 0

VMware vCloud Director Resource Allocation Models

Test Cases For the purposes of the following examples two test cases will be used and specific conclusions will be presented

bullThreecustomerswithasingleallocationmodeltypebutwithdifferentsettingsappliedtoeachsharingasingle provider VDC

bullThreecustomerswithdifferentvCloudallocationmodelseachsharingasingleproviderVDC

A single provider VDC will be mapped to the vSphere DRS cluster of the test environment A standard vApp is deployed containing two virtual machines The configuration of the provider VDC and the vApp templates are listed in Table 11

SETTING CPU MEMORY

Provider VDC 15GHz 50GB

Virtual machine 1 2 vCPU 4096MB

Virtual machine 2 2 vCPU 4096MB

Memory overhead reservation virtual machine

250MB

Table 11 Test vCloud Environment Configuration

In all tests each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and are running 100 percent workload The total required memory demand will exceed the current capacity of 50GB offered by the provider VDC

NOTEThescenariosareprimarilyfocusedonmemoryresourcemanagement

Single Allocation Model TestsSingle Allocation Model ndash Pay-As-You-Go

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 026GHz per vCPU 0 100 100

Customer 2 026GHz per vCPU 0 100 50

Customer 3 026GHz per vCPU 0 100 0

Table 12 Pay-As-You-Go Virtual Machine Resource Settings Overview

The pay-as-you-go model retains the expandable reservation setting at the resource pool level enabling each customer to withdraw resources from the provider VDC However because each customer has different settings the potential performance for each customerrsquos virtual machines might vary All virtual machines in customer 1rsquos organization will be allocated 100 percent of their physical resources customer 3rsquos organization might use all of its virtual resources and very little of its physical resources if contention occurs

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 31: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 3 1

VMware vCloud Director Resource Allocation Models

Reserved Memory

17384MB 9192MB 8192MB 15342MB 1000MB

Unreserved Memory

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand

Provider VDC

17384MB Demand

Figure 29 Resource Distribution Pay-As-You-Go Scenario

Customer 1 The virtual machines of customer 1 receive all their required resources because 100 percent of the virtual machine memory is guaranteed

Customer 2 The virtual machines of customer 2 also receive their required resources but the remaining 8192MB is based on opportunistic allocation

Customer 3 Allocation of resources by the virtual machines of customer 3 is solely based on opportunistic allocation Figure 26 shows a small portion of guaranteed resources allocated by the virtual machine memory overhead reservation of the virtual machines The amount of memory allocated by customer 3 is 16342MB resulting in 952MB of memory reclamation among the virtual machines in the organization VDC

The interesting part of this scenario is the process of dividing the unreserved memory between the organization VDCs of customer 2 and customer 3

bullThevirtualmachinesofcustomer1receivetheirfullallocation

ndash 17184MB of reserved resources are allocated

ndash The unreserved pool in the provider VDC contains 34016MB of unreserved memory

bullTheallocationmodeloftheorganizationVDCofcustomer2isconfiguredwitha50percentmemoryguarantee

ndash 8192MB of reserved resources are allocated

ndash There is an additional 1000MB of memory overhead reservation

ndash The total reserved memory allocation is 9192MB

ndash The unreserved pool in the provider VDC contains 24824MB of unreserved memory

bullAfterpower-onofthevirtualmachinesofcustomer3

ndash 1000MB are withdrawn from the provider VDC unreserved memory pool

ndash The unreserved pool in the provider VDC contains 23624MB of unreserved memory

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 32: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 3 2

VMware vCloud Director Resource Allocation Models

8192MB

50Priority

50Priority

11812MB 11812MB

8192MB16384MB 15432MB

Provider VDC

23624MB 23624MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 30 Distribution of Unreserved Memory Resources in Pay-As-You-Go Test

The remaining 23624MB of unreserved memory is divided between the resource pools of customer 2 and customer 3 because the pay-as-you-go model leaves the resource pool settings default The result is that the two resource pools are configured with the same share value

During contention the two resource pools receive the same priority to access memory

bullStep1Bothresourcepoolsareallowedtoclaim11812MB(23624MBdividedby2)ofmemory

bullStep2Theresourcepoolofcustomer2requiresonly8192MB(17384MBminus9192MB)itclaimsthe8192MB leaving 3620MB in the pool

bullStep3Thevirtualmachinesofcustomer3demandmorethantheallocated11812MB(17384MB)resourcepool 3 is able to claim the available 3620MB

In the previous scenario all vApps and their virtual machines were powered on simultaneously Each customer had exactly the same number of vApps and associated virtual machines with identical configurations something that would not occur very often in a real-life scenario If customer 1 decides to power on another vApp admission control will allow it because there are enough unreserved resources available in the provider VDC Because the allocation model of customer 1 is configured with a 100 percent memory guarantee the virtual machines will perform as well as the previous powered-on virtual machines However the virtual machines of customer 2 and customer 3 likely will experience performance degradation as memory is reclaimed on behalf of virtual machines of customer 1 Customer 2 will continue to have access to 9192MB of memory due to its memory guarantee customer 3 potentially can lose access to up to 1000MB of physical memory

Design consideration The pay-as-you-go model does not alter resource pool default settings and does not set a reservation and limit at this level resulting in an ldquoopenrdquo architecture The resource pool does not offer a pool of guaranteed resources that virtual machines can consume when required This leaves the virtual machines vulnerable to the activity of other virtual machines external to the resource pool

Access to physical memory depends on the guaranteed memory configuration and the activity and configuration of peer organization VDCs sharing the same provider VDC Keep this in mind when deploying multiple pay-as-you-go allocation models on one provider VDC

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 33: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 3 3

VMware vCloud Director Resource Allocation Models

Single Allocation Model ndash Reservation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 Equal to reservation 3000MHz Equal to reservation 20GB

Customer 2 Equal to reservation 3000MHz Equal to reservation 15GB

Customer 3 Equal to reservation 3000MHz Equal to reservation 10GB

Table 13 Reservation Pool Resource Pool Resource Settings Overview

As we have seen the reservation pool allocation model preallocates a set of physical resources on resource pool level The resource consumption of this allocation model is easy to predict because it does not leverage burst space of the provider VDC The organization VDC is configured with a guaranteed amount of reserved resources and cannot exceed this because the limit is automatically set equal to the guaranteed resources This characterizes this allocation model as a ldquoconfiningrdquo structure

Overallocation of resources can occur within the resource pool because no virtual machine reservations are set by default

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC Due to the confining element no resource demand will traverse resource pool boundaries Instead it will result in memory reclamation within the resource pool

Customer 1 Customer 2 Customer 3

17384MB Demand 17384MB Demand 17384MB Demand

Reserved Memory

Allocated MemoryNot Used

Unavailable Memory

20480MB 15360MB 10240MB

Provider VDC

Figure 31 Resource Distribution Reservation Pool Scenario

Customer 1 The resource supply of the organization VDC of customer 1 does not exceed the resource demand of the virtual machines running within the resource pool Therefore all virtual machines receive 100 percent of their required resources

Customer 2 The resource demand of the virtual machines of customer 2 exceeds the resource supply and consequently must share the allocated physical memory Due to the configured limit the resource pool is unable to allocate more physical memory and the excess memory demand of 2024MB is reclaimed from the running virtual machines Because all four virtual machines are configured identically and have received the same amount of shares memory reclamation is done equally on all four virtual machines

Customer 3 Virtual machines running in the organization VDC of customer 3 face a similar situation Due to the mismatch of resource supply and demand memory will be reclaimed among the virtual machines The virtual memory overhead reservation is withdrawn from the resource pool memory resources during power-on operations extracting a total of 1000MB of the available 10240MB before the guest operating system (OS) and workloads can allocate the remaining memory

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 34: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 3 4

VMware vCloud Director Resource Allocation Models

Memory OverheadReservation

VM1

4GB3560

0

2390

0

3GB

2GB

1GB

0

VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2 VM1 VM2

20480MB Available

vApp1

MemoryReclamation

Customer 1

vApp2

15360MB Available

vApp1

Customer 2

vApp2

10240MB Available

vApp1

Customer 3

vApp2

Figure 32 Distribution of Memory Resources Within Resource Pools

Design consideration The reservation pool model confines resource demand by allocating and limiting physical resources on a resource pool level Resource allocation settings by default are not configured on virtual machines deployed in the organization VDC enabling overconsolidation within the organization VDC This assumes the original template used had properly leveled per virtual machine settings

Overallocation of compute resources within one VDC does not impact workloads of other organization VDCs However if overconsolidation on memory takes on extreme forms memory swapping might impact storage layer performance of virtual machines external to the organization VDC as vCloud Director distributes virtual machines across available datastores within the provider VDC

Single Allocation Model ndash Allocation Pool

SETTING CPU LIMIT CPU RESERVATION

MEMORY LIMIT MEMORY RESERVATION

Customer 1 3GHZ 0 20GB 20GB (100)

Customer 2 3GHZ 0 20GB 15GB (75)

Customer 3 3GHZ 0 20GB 10GB (50)

Table 14 Allocation Pool Resource Settings Overview

The allocation model is a hybrid model combining traits of the pay-as-you-go and reservation pool models The main difference between the pay-as-you-go and allocation pool models is the limit setting on the resource pool of the allocation pool model The limit restricts the resource pool in allocating physical memory above the configured threshold which makes the allocation of physical resources relatively easy to predict However if the default reservation setting is decreased from 100 percent to a custom setting the organization VDC can allocate resources from the burst space of the provider VDC

Decreasing the default setting of 100 percent guaranteed resources does not enable overallocation of resources as described in the paragraphs under the ldquoReview of Resource Distributionrdquo section ldquoAllocation Pool Modelrdquo subsection Reducing the default percentage of 100 percent only increases the amount of opportunistic resource allocation which enables the cloud provider to increase the consolidation ratio of organization VDCs per provider VDC

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 35: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 3 5

VMware vCloud Director Resource Allocation Models

The unreserved resources belonging to the provider VDC function as burstable space for the allocation pool Organization VDCs that are configured for opportunistic allocation can compete for the unreserved resources

The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps The resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 Resource demand of the virtual machines of customer 2 exceeds the amount of available guaranteed resources leading to opportunistic allocation of 2024MB (17384MB demand minus 15360MB supply)

Customer 3 Resource demand of the virtual machines of customer 3 exceeds the amount of available guaranteed resources as well leading to opportunistic allocation of 7144MB (17384MB demand minus 10240MB supply)

Customer 1(20GB Limit)

Customer 2(20GB Limit)

17384MB Demand

Customer 3(20GB Limit)

17384MB Demand

17384MB Demand

Provider VDC

Allocated MemoryNot Used

20480MB 15360MB 5120MB 10240MB

Figure 33 Resource Distribution Allocation Pool Scenario

Both resource pools compete for the required resources with their resource pool settings In total 46080MB of memory is allocated to the resource pools leaving 5120MB unreserved This 5120MB is the burst space of the provider VDC and is the arena where resource pools of both customers 2 and 3 compete for resource allocation The total amount of required resources is 9168MB The amount of memory resources available is 5120MB resulting in an oversubscription rate of 179 High oversubscription rates can lead to a performance decrease in virtual machines The oversubscription rate of the burst space should align with the service-level agreements

Unreserved Physical Resources

5120MB

2024MB 7144MB

Provider VDC

Customer 2Demand

Customer 3Demand

Figure 34 Oversubscription of Burst Space

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 36: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 3 6

VMware vCloud Director Resource Allocation Models

The impact of the oversubscription on the burst space can have the following effect if all virtual machines are equally active and demand resources

EXAMPLEResourcepoolsofcustomers2and3arerequiredtoallocateresourcesfortheirvirtualmachinesTheresourcepoolofcustomer3demandsmorememorythantheresourcepoolofcustomer2HowevertherelativesharevalueoftheresourcepoolwilldeterminetheallocationofresourcesThefollowingpointsarevalidbecausebothresourcepoolsareconfiguredwiththe(default)normalsharevalue

1 The two resource pools will receive equal priority

2 The equal priority enables both resource pools to claim up to 2560MB of memory However because resource pool 2 requires only 2024MB it claims the required 2024MB leaving 536MB in the pool

3 The virtual machines within the resource pool of customer 3 demand more than the allotted 2560MB Resource pool 3 is able to claim the available 3096MB for its use

2024MB

50Priority

50Priority

2560MB 2560MB

2024MB7144MB 3096MB

Provider VDC

5120MB 5120MB

Provider VDC

1 2

3

Customer 3Normal Shares

Customer 2Normal Shares

Customer 3Normal Shares

Customer 2Normal Shares

Figure 35 Distribution of Unreserved Memory Resources in Allocation Pool Test

Due to the imbalance of resource demand versus resource supply of virtual machines within the resource pool of customer 3 virtual machines of customer 3 might be subjected to memory reclamation possibly impacting performance

Design consideration The allocation pool model configures reservations and limits on both virtual machine level and resource pool level Reducing guarantees does not alter the consolidation ratio of virtual machines to organization VDCs but it does provide a higher consolidation ratio of organization VDCs to provider VDCs A key design consideration is the availability of burst space and the desired oversubscription rate A higher oversubscription rate can lead to unavailability of resources for some organization VDCs which can impact the performance of virtual machines Therefore oversubscription should align with the service-level agreements

Multiple Allocation Models Two tests have been conducted to illustrate the operation of resource allocation when multiple allocation models are used within one provider VDC The default settings are applied in the first test The second test illustrates the impact of mixed allocation models also configured with custom settings The settings used are set to 75 percent of the default numbers

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 37: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 3 7

VMware vCloud Director Resource Allocation Models

Default Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 20GB

Customer 2 Allocation pool 3GHz 0 20GB 100

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 100

Table 15 Multiple Allocation Default Resource Settings Overview

The first test focuses on using different allocation models with the default settings As in previous tests this one mainly focuses on resource allocation of memory The provider VDC contains 50GB of unreserved memory as it did before First the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 20GB of memory of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 30GB unreserved memory

Customer 1

Provider Virtual Datacenter

20480MBGuaranteed

Figure 36 Organization VDC Customer 1

Next the organization VDC for customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 100 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 20GB reservation and a 20GB limit The resource pool instantly claims the 20GB of reserved memory leaving the provider VDC with 10GB unreserved memory

Customer 1 Customer 2

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

Figure 37 Organization VDC Customer 2

In the previous step the organization VDC for customer 3 was created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation or limit settings of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 38: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 3 8

VMware vCloud Director Resource Allocation Models

Customer 1 Customer 2 Customer 3

Provider Virtual Datacenter

20480MBGuaranteed

20480MBGuaranteed

0MBGuaranteed

Figure 38 Organization VDC Customer 3

After all three organization VDCs are created each customer powers on two vApps containing two 4GB virtual machines each All vApps are powered on at the same time and run 100 percent workloads The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 contains enough resources to satisfy the resource demand of the two vApps Similar to the previous tests the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Customer 2 The resource pool for the organization VDC of customer 2 also contains enough resources to satisfy the resource demand of the two vApps Similar to previous tests and to the performance of the vApps and resource pool of customer 1 the resource demand of 17384MB does not reach the limit of 20480MB And because 100 percent of the resources are guaranteed all memory consumed by the virtual machines is backed by physical memory

Although customer 2 has powered on as many vApps as customer 1 customer 2 can no longer power on a single vApp containing two virtual machines At this point 17384MB is withdrawn from the resource pool and distributed exclusively to the powered-on virtual machines to satisfy their virtual machinendashlevel reservation The resource pool contains 3096MB of memory not enough for admission control to allow powering on another vApp that requires 8692MB

The performance of the reservation pool and allocation pool is quite similar from a cloud provider perspective Both models set reservations at the resource pool level and both models immediately claim resources and reduce the available resources on the provider VDC level The only difference is the consolidation ratio of virtual machines to organization VDCs If the two models are configured with the same memory allocation it can result in an increase in load of the storage platform if the customer is overallocating memory within the resource pool

Customer 3 Resource pools of customers 1 and 2 are configured with a limit and the remaining resources in the provider VDC are available to customer 3 That available memory in this scenario is 10GB The pay-as-you-go model sets a 100 percent reservation on the virtual machine level Therefore customer 3 can power on only one vApp with two virtual machines because admission control can satisfy a request of only 8692MB Due to the virtual machinendashlevel reservation no memory overcommitment of available resources of the provider VDC is possible

Custom Resource Allocation Settings Test

SETTING ALLOCATION MODEL

CPU LIMIT CPU RESERVATION

MEMORY LIMIT

MEMORY RESERVATION

Customer 1 Reservation pool 3GHz 100 100 15GB

Customer 2 Allocation pool 3GHz 75 20GB 75

Customer 3 Pay-as-you-go 026GHz per vCPU

0 100 75

Table 16 Multiple Allocation Custom Resource Settings Overview

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 39: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 3 9

VMware vCloud Director Resource Allocation Models

The second test focuses on using different allocation models along with the custom settings In many ways this is what customers are asking for as ldquoTrue Cloudrdquo Each setting is adjusted to 75 percent of the previous setting The guaranteed memory setting of the organization VDC of customer 1 is decreased from 20GB to 15GB The limit of customer 2 remains at 20GB however the guaranteed resource setting is decreased to 75 percent The pay-as-you-go allocation model applied to the organization VDC of customer 3 is reduced from 100 percent to 75 percent

As in previous tests this one focuses mainly on resource allocation of memory The provider VDC contains 50GB of unreserved memory As before first the organization VDC of customer 1 is created This organization VDC uses a reservation pool model and is set to allocate 15GB of guaranteed physical resources vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 15GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 35GB of unreserved memory

Customer 1

Provider VDC

15360MBGuaranteed

Figure 39 Organization VDC Customer 1

Next the organization VDC of customer 2 is created It is configured with the allocation pool model The default settings are used and 20GB of memory is allocated with a 75 percent guarantee vCloud Director instructs vCenter Server to spawn a resource pool configured with a 15GB reservation and a 20GB limit The resource pool instantly claims the 15GB of reserved memory leaving the provider VDC with 20GB of unreserved memory The difference between the reservation and the limit that can be allocated by the resource pool is based on the share value and the activity of the external organization VDCs

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed

Figure 40 Organization VDC Customer 2

Similar to the last test the organization VDC of customer 3 is created and configured with a pay-as-you-go model Because the pay-as-you-go model sets virtual machines reservation a resource pool at vSphere layer is created without adjusting the reservation and limit setting of the resource pool This results in an empty resource pool without impact resource availability at the provider VDC level However in this scenario an overlap of possible resource allocation is created The resource pool belonging to customer 2 is competing for unreserved resources with the resource pool of customer 3

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 40: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 4 0

VMware vCloud Director Resource Allocation Models

Customer 1

Provider VDC

15360MBGuaranteed

Customer 2

15360MBGuaranteed Customer 3

0MBGuaranteed

Figure 41 Organization VDC Customer 3

After all three organization VDCs have been created each customer powers on two vApps each containing two 4GB virtual machines All vApps are powered on at the same time and run a 100 percent workload The total required memory demand exceeds the current capacity of 50GB offered by the provider VDC

Customer 1 The resource pool of the organization VDC of customer 1 does not contain enough physical resources to satisfy the resource demand of the two vApps The resource demand of 17384MB exceeds the limit of 15360MB Due to the configured limit the resource pool is unable to allocate more physical memory if necessary even when enough resources are available in the provider VDC

Due to the absence of virtual machinendashlevel memory reservations each virtual machine allocates its required resources on priority based on its relative share level Each virtual machine is configured identically and is 100 percent utilized resulting in an equal distribution of resources among the virtual machines In this scenario each virtual machine is entitled to 3560MB resulting in a 506MB memory reclamation of each virtual machine Due to the 100 percent utilization of each virtual machine within the resource pool memory reclamation will be provided by either ballooning unused memory within the virtual machine compressing memory or allocating virtual memory by moving out pages to a virtual machine swap file located on a datastore

Customer 2 The resource pool of the organization VDC of customer 2 can allocate enough physical resources to satisfy the resource demand of the two vApps However 2024MB is based on opportunistic access The resource pool disposes over 15360MB of reserved physical resources The remaining 204MB of the 17384MB resource demand fits within the limit of the resource pool enabling the resource pool to acquire physical resources to satisfy demand The ability to acquire resources by the resource pool is based on the share value and the activity of the external resource pools belonging to other organization VDCs

Customer 2

Provider VDC

15360MBGuaranteed

5120MBOpportunistic

Allocation

Figure 42 Opportunistic Resource Allocations

Although reducing the percentage of guaranteed resources on the allocation pool model both virtual machinendash and resource poolndashlevel reservations are decreased However this does not result in the ability to power on more virtual machines

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 41: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

T E C H N I C A L W H I T E P A P E R 4 1

VMware vCloud Director Resource Allocation Models

In this scenario 75 percent of the configured memory of the virtual machine is guaranteed The resource pool is required to satisfy the virtual machine reservation of 3072MB (75 percent of 4096MB) per virtual machine Because each virtual machine has a memory overhead reservation of 250MB a total of 3322MB (3072MB plus 250MB) of memory is required per virtual machine The vApp consists of two virtual machines resulting in a total memory reservation of 6644MB (3322MB plus 3322MB) per vApp The two vApps consume 13288MB (6644MB plus 6644MB) leaving the resource pool of 2078MB of available reserved resources not enough for admission control to allow powering on another vApp that requires 6644MB

Customer 3 Due to the ability of the resource pool of customer 2 to allocate memory from the burst space virtual machines of customer 3 can compete for resources with virtual machines from customer 2 The virtual machines of customer 1 are confined within the resource pool and will not allocate memory resources from the unreserved pool of the provider VDC The burst space of the provider VDC contains 20GB of unreserved memory

In this scenario the pay-as-you-go model sets a 75 percent reservation on the virtual machine level Therefore customer 3 can power on three vApps combining for a total requirement of 19932MB of memory This fits within the 20480MB burst space of the provider VDC The resulting 548MB are distributed equally among the resource pools of customers 2 and 3 because the two resource pools are configured with normal share values and experience equal workloads

NOTEIfavirtualmachineorresourcepoolallocatesmemorybasedonsharevaluethismemorycanbereclaimedifavirtualmachineispoweredonwithavirtualmachinendashlevelreservationSimilarsituationsoccurwhenaresourcepooliscreatedwithamemoryreservationconfigured

About the AuthorsFrank Denneman is senior architect in the Technical Marketing group at VMware His focus is on resource management and downtime avoidance with a specialty in vSphere DRS vSphere Storage DRS VMware vSphere vMotionreg and VMware vSphere Storage vMotionreg He is the co-author of multiple books including best sellers vSphere50ClusteringTechnicalDeepdive and VMwarevSphere41HAandDRSTechnicalDeepdive Frank is the owner and main author of one of the top virtualization and VMware blogs frankdennemannl He is among the first 50 VMware Certified Design Experts (VCDX29)

bullFollowFrankrsquosblogsathttpfrankdennemannl and httpblogsvmwarecomvSphere

bullFollowFrankonTwitterfrankdenneman

Chris Colotti is a consulting architect with the VMware Center of Excellence He has more than 10 years of experience working with IT hardware and software solutions He holds a Bachelor of Science degree in Information Systems from the Daniel Webster College Prior to coming to VMware he served a Fortune 1000 company in southern New Hampshire as a systems architectadministrator architecting VMware solutions to support new application deployments At VMware Chris has guided partners as well as customers in establishing a VMware practice and has consulted on multiple customer projects ranging from datacenter migrations to long-term residency architecture support Chris is also among the first 50 VMware Certified Design Experts (VCDX37)

bullFollowChrisrsquoblogathttpwwwchriscolottius and httpblogsvmwarecomvCloud

bullFollowChrisonTwitterCColotti

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01

Page 42: VMware vCloud Director Resource Allocation Models€¦ · Director resource allocation models. These are represented in vSphere by “resource pools,” defined in Table 2. Resource

VMware Inc 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www vmware comCopyright copy 2012 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patents listed at httpwww vmware comgopatents VMware is a registered trademark or trademark of VMware Inc in the United States andor other jurisdictions All other marks and names mentioned herein may be trademarks of their respective companies Item No TBD Docsource OIC - 12VM013 01