Understanding Host and Guest Memory Usage and Other Memory Management Concepts
-
Upload
bmdfljgmdfjgmlfkgm -
Category
Documents
-
view
738 -
download
3
Transcript of Understanding Host and Guest Memory Usage and Other Memory Management Concepts
Understanding “Host” and “Guest” Memory Usage and Related Memory Management Concepts
Kit ColbertSr. Staff Engineer, VMware
DisclaimerThis session may contain product features that are currently under development.
This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product.
Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new technologies or features discussed or presented have not been determined.“These features are representative of feature areas under development. Feature commitments are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery.”
Agenda
Motivation
Define host and guest memory usage and ask some questions
Memory management concepts
Answer our questions
Best practices
Summary
Q & A
Motivation
Host and Guest Memory Usage
Using Host and Guest Memory Usage
Useful for quickly analyzing a VM’s status– Coarse-grained information
– Important for prompting further investigation
Requires an understanding of memory management concepts– Many aspects of host/guest memory interaction are not obvious
Define host and guest memory and ask some questions
Host Memory Usage Defined
Host Memory Usage– The amount of physical host memory in megabytes allocated to a
guest• Includes virtualization overhead (e.g. hypervisor data)
Guest Memory Usage Defined
Guest Memory Usage– The amount of memory in megabytes actively used by a guest
operating system and it’s applications
– The percent of total guest memory actively used by a guest
Memory Terminologymemory size
total amount of memory presented to a guest
allocated memorymemory assigned to
applications
unallocated memorymemory not assigned
active memoryallocated memory recently
accessed or used by applications
inactive memoryallocated memory not recently accessed or
used
Guest memory usage measures this
Host memory usage measures this, sorta…
Some commonly asked questions
Why is host memory usage so high?
Why is host memory usage greater than guest memory usage?
Why is host/guest memory usage different than what I see inside the guest?
Memory Management Concepts
Discussion of Concepts
Focus on high-level concepts, not implementation details– Simpler and easier to understand
– Keep the fundamental ideas and drop the unimportant ones
Other presentations may present this information differently– May discuss more implementation than concept
– That’s ok – different viewpoints for same thing!
Virtual Memory
Creates uniform memory address space– Operating system maps application virtual
addresses to physical addresses
– Gives the operating system memory management abilities that are transparent to the application
“virtual” memory
“physical” memory
“machine” memory
guest
hypervisorHypervisor adds extra level of indirection
– Maps guest’s physical addresses to machine addresses
– Gives the hypervisor memory management abilities that are transparent to the guest
Virtual Memory
guest
hypervisor
“machine” memory
“physical” memory
“virtual” memory
“virtual” memory
“physical” memory
“machine” memory
guest
hypervisor
Application
Operating System
Hypervisor
App
OS
Hypervisor
VM
Application Memory Management
Starts with no memory
Allocates memory through syscall to operating system
Often frees memory voluntarily through syscall
Explicit memory allocation interface with operating system
Hyper visor
OS
App
Operating System Memory Management
Assumes it owns all physical memory
No memory allocation interface with hardware
– Does not explicitly allocate or free physical memory
Defines semantics of “allocated” and “free” memory
– Maintains “free” list and “allocated” lists of physical memory
– Memory is “free” or “allocated” depending on which list it resides
Hyper visor
OS
App
How “Allocated” and “Free” Lists Work
OS
AppAllocated List Free List
Memory becomes “allocated” or “free” based on which list (piece of memory) has a pointer to it
Hypervisor Memory Management
Very similar to operating system memory management
– Assumes it owns all machine memory
– No memory allocation interface with hardware
– Maintains lists of “free” and “allocated” memory
Hyper visor
OS
App
VM Memory Allocation
VM starts with no physical memory allocated to it
Physical memory allocated on demand
– Guest OS will not explicitly allocate
– Allocate on first VM access to memory (read or write)
Hyper visor
OS
App
VM Memory Reclamation
Guest physical memory not “freed” in typical sense
– Guest OS moves memory to its “free” list
– Data in “freed” memory may not have been modified
Hyper visor
OS
App
Hypervisor isn’t aware when guest frees memory
– Freed memory state unchanged
– No access to guest’s “free” list
– Unsure when to reclaim “freed” guest memory
Guest Free List
Guest OS (inside the VM)– Allocates and frees…
– And allocates and frees…
– And allocates and frees…
Hyper visor
AppGuestfree list
VM– Allocates…
– And allocates…
– And allocates…
Hypervisor can’t reclaim memory through guest frees!
Inside the VM
OS
VM
VM Memory Reclamation Cont’d
What to do about VM memory reclamation?
Why not just watch the free list?
Idea– Hypervisor could identify memory used by guest for it’s free list
– Watch that memory area for changes and free physical memory that backs guest memory added to free list
Problem– Free list may be hard to identify
• Depends on guest type and may change from release to release
– “Free” is not always free• “Free memory” may be used in buffer cache or elsewhere by guest
– Guest may not free memory when hypervisor wants it to
What can the hypervisor do to reclaim VM memory?
Nothing– Route taken by pretty much everyone except VMware
– No memory overcommitment! (Inefficient use of memory)
– May try to convince you memory overcommitment isn’t important
Something– VMware-innovated techniques
– Supports memory overcommitment• Efficient use of memory
• Strict resource control guarantees
Memory Overcommitment
Aggregate guest memory greater than host memory– Not enough physical memory to satisfy all VM memory needs
– Eventually VM may want physical memory when none is left
VM 1 VM 2 VM 3
Hyper visor
Why is memory overcommitment important?
Ensures physical memory is actively used as much as possible– Guest VMs may have lots of unused or inactive memory
– No reason to back one VM’s unused/inactive memory with physical memory if other VMs can use it
– Hypervisor can reclaim unused/inactive memory and redistribute to other VMs who will actively use it
Increases VM-to-host consolidation ratio– Each VM has a smaller physical memory footprint since only its
active memory is resident
– You can fit more VMs in the same amount of physical memory
VM Memory Reclamation Techniques
Simple idea: why maintain many copies of the same thing?
– If 4 Windows VMs are running, there are 4 copies of Windows code
– Only one copy is needed
Share memory between VMs when possible
– Background hypervisor thread identifies identical sets of memory
– Points all VMs at one set of memory, frees the others
– VMs are unaware of the change
VM 1 VM 2 VM 3
Hyper visor
VM 1 VM 2 VM 3
Hyper visor
Transparent Page Sharing (TPS)
Hypervisor wants to reclaim memoryGuest OS is not aware of this
– Thinks it owns all physical memory– Sits inside its own “box”, unaware it’s
running in a VM or that other VMs are running
Goal: make the guest aware so it frees up some of its memory
Hyper visor
OS
App
OS
App
VM 1
Solution: artificially create memory pressure inside the VM
– “Push” memory pressure from the hypervisor into the VM
– Use “balloon” driver inside the VM to create memory pressure
VM 2
memory pressure
Ballooning
Hyper visor
OS
GuestApp
Guestfree list
BalloonDriver
1. Balloon driver allocates memory
2. Balloon driver pins allocated memory
3. Guest may reclaim other memory
4. Balloon driver tells hypervisor what memory it allocated
5. Hypervisor frees machine memory backing memory allocated by balloon driver
6. Hypervisor now has more free physical memory
List of memory
Hypervisor: 8 pieces of memory are allocated
Guest: 3 pieces of memory are allocated
Hypervisor: 5 pieces of memory are allocated
Guest: 6 pieces of memory are allocated
Inflating Balloon
Hyper visor
OS
BalloonDriver
Why can the hypervisor reclaim ballooned memory?
1. Balloon driver doesn’t
2. OS doesn’t
3. Nothing in the VM relies on the memory’s value for correctness
What in the VM assumes the memory has a specific value?
Because of a contract with the hypervisor
Because the memory allocated to an app List of
memory
Hypervisor can safely reclaim ballooned memory because the VM does not rely on a particular value for that memory.
Inflating Balloon Cont’d
Hyper visor
OS
GuestApp
Guestfree list
BalloonDriver
Guest OS swapping, a possible side effect of ballooning Two possibilities for guest free memory:
2. VM doesn’t have much free memory
1. VM has lots of free memory
Guestfree list
1. No swapping necessary!
2. Needs to swap!
Guest OS chooses whether to swap or not!
Inflating Balloon Cont’d
Hypervisor can swap VM memory– Swaps out to a per-VM swap file
– Transparent to the VM
Hypervisor swapping is a last resort– Both TPS and ballooning preferred
• Low overhead for TPS
• Guest swapping due to ballooning performs better than hypervisor swapping
• But both take time
– Hypervisor swapping quickly reclaims memory, but is more expensive overall
Hyper visor
OS
App
Swapping
When to reclaim memory
Quick Review
Physical memory is not reclaimed on a guest that is “free”– VM may accrue lots of physical memory
Transparent page sharing is always running– May or may not help reduce a VM’s physical memory consumption
Ballooning or swapping is the only other way to reclaim memory– Both have performance overhead
– Want to use only when necessary
Question: when to invoke ballooning or swapping?
When To Reclaim Memory
Memory overcommitment– Aggregate guest memory is greater than host memory
Not memory overcommitted– Never reclaim memory from guest through ballooning or swapping
• No need to!• (Transparent page sharing is always running)
Memory overcommitted– Only reclaim once free physical memory drops below threshold
• Start ballooning when memory starts to fall toward threshold• Start swapping as memory nears/passes threshold
* Assumes no VM or resource pool memory limits are set!
Answer our questions
High Host Memory Usage
Why is host memory usage so high?– VM may have used lots of memory in the past
• On boot, Windows zeros all guest memory
• Starting several guest applications together, e.g., at boot time or when building a big project, can lead to high memory usage
• Guest workload spike
– Host memory usage represents “high water mark” of guest’s memory usage on non-memory overcommitted hosts
• No reason to reclaim memory from VM when not memory overcommitted
Host Memory Greater Than Guest Memory
Why can host memory be greater than guest memory?– At some point in the past, the guest consumed a larger amount of
memory than it is actively using now
– With little to no memory pressure on the physical host, there is no reason for hypervisor to reclaim memory from the guest
– The guest maintains a high host memory usage even though its guest memory usage is small
Expected behavior– Nothing to worry about!
Host/Guest Memory Usage Different Inside Guest
Why is host/guest memory usage different than what I see inside the guest OS?
– Guest memory• Guest has better visibility while estimating “active” memory
• ESX active memory estimate technique can take time to converge
– Host memory• Host memory usage doesn’t correspond to any memory metric within the
guest
• Host memory usage size is based on a VM’s relative priority on the physical host and memory usage by the guest
Again, this is expected!
Best Practices
Host Memory Usage
Large host memory usage OK– Expected behavior much of the time!
Host memory usage shouldn’t limit guest memory usage– Host memory usage should be large enough to accommodate guest
workload
– Don’t over commit host memory forcing the guest to continuously swap
Use shares to adjust relative priorities among VMs when memory is overcommitted
– Host memory usage is based on shares when overcommitted
Guest Memory Usage
VM’s memory size should be slightly larger than the average guest memory usage
– Accommodates workload spikes without guest swapping
– Hypervisor will handle excess host memory usage
– Larger VM memory size means more overhead memory
Summary
Summary
Host/guest memory usage is not as simple as it looks!– Represents complex set of data
– Requires understanding of memory management concepts
– Many aspects that are not obvious!
Memory Management Concepts– Guest doesn’t “free” memory in a typical sense
– Hypervisor isn’t aware of memory a guest has “freed”
– Hypervisor uses transparent page sharing, ballooning, and swapping to reclaim memory
– Ballooning and swapping used only when host is memory overcommitted!
Summary Cont’d
Host/guest memory can be very useful with proper knowledge– Quick way to determine the status of VM
– Important tool to prompt further investigation
– Requires good mental model of memory management concepts• Hopefully we’ve helped you with that today!
Q&AKit ColbertSr. Staff Engineer VMware
Thank you for coming.
Rate your session and watch for the highest scores!
Backup Slides
Measuring Active Guest Memory
Guest self-measurement won’t work– Each guest uses different method to estimate active memory
– Can’t compare activeness data from Windows guest with Linux guest
– Comparable estimate important for making allocation decisions
Statistical Sampling– Select a subset of memory at random
– Compute the percentage of pages accessed in a minute
– Average percentage gives an estimate of “active” guest memory
Ballooning and Swapping
Ballooning– Provide adequate swap space in guest OS
• Ballooning can lead to high guest memory swapping
• Ballooning not effective if a guest can’t swap
– Limit maximum balloon size if lots of guest memory pinned• Pinned memory cannot be swapped
• Use sched.mem.memctl.max to set maximum balloon size
Swapping– Ensure adequate swap space on VMFS datastore
• Swap size = VM memory size – VM memory reservation