Virtual Memory Management: Dynamic Memory-Much Different Than Memory Over Commit – Become a Virtualization Expert (Part 3 of 20)

Hyper-V 3 which is built into both the Free Hyper-V Server 2012 and Windows Server 2012 (Standard and Datacenter) which you can evaluate for free (free ISO, free VHD), has some really awesome capabilities. You can read more about Hyper-V in the overview of the Server Virtualization Series: Hyper-V Basics. In this article, we are going to drill down into what I think is one of the best features of Hyper-V. A feature called Dynamic Memory. Dynamic memory has significantly changed the world of server virtualization. As part of the discussion, we will also cover memory virtualization on other platforms so you get a solid understanding of the very, very important concept of “Virtual Memory”. In addition in keeping with the theme of the Server Virtualization Series we will sprinkle in Step-By-Step & How-To’s so you can do it yourself. Dynamic Memory is an optional feature that you can turn on and off with a simple checkbox. With Dynamic Memory you can attain higher consolidation numbers with improved reliability for restart operations. This can lead to lower costs, especially in environments that have many idle or low-load virtual machines, such as pooled VDI environments. Dynamic Memory run-time configuration changes can reduce downtime and provide increased agility to respond to requirement changes. For more on runtime configuration changes see Kevin Remde‘s blog post You Want to Hot-Add What?! : 20+ Days of Server Virtualization (Part 2 of 20). If you have not yet installed Hyper-V you can follow the Hands-On-Lab guide at http://itproguru-app.azurewebsites.net/hol it is the “Install/Enable and Configure Hyper-V Getting Started” lab.

Inside Dynamic Memory

What is Dynamic Memory?

Dynamic memory was introduced in Windows Server 2008 R2 with SP1. It gave the Hyper-V hypervisor the capability to dynamically adjust the amount of memory assigned to a machine based on memory demand. The reason we would want to do this is simple. If we can put only the amount of memory a machine needs in the box, we can fit more machines on a host. How’s that? When you deploy a server you give it the amount of memory you think it will need at it’s maximum load. So if when the server is the busiest it needs 4gb of memory it will have what it needs. But, what if that server is only busy for an hour a week? That is a waste of the 4gb of memory. Dynamic memory allows Hyper-V to “Dynamically Reallocate” that memory to other servers that may need it. Then when it needs the memory it can simply dynamically reallocate it back.

An important concept that you need to know about dynamic memory is that it is not the same as “memory overcommit.” Dynamic memory is a much more accurate term to use for it because Hyper-V dynamically “allocates” memory based on workload “demand” In this article, all of these terms and distinctions will be clearly defined. Hyper-V DOES NOT OVERCOMMIT! One of the biggest problems with “memory overcommit” is what happens under memory pressure. When a server runs out of memory it pages (swaps to disk). This is a very costly (slow performance) transaction so Hyper-V will not page at the host (with one very small exception shown below). If there is going to be paging it has to be at the guest and only on the lower priority servers.

Dynamic memory is constantly managing memory. Some call this a disadvantage because it is always working. I do not see it that way. I actually see it as a huge advantage. First, the probability of getting in a low memory state is much lower because Hyper-V is constantly watching demand and tweaking things for optimum performance. Second, when there is memory pressure it does not have to do all that extra work to sort things out while in a starving situation. Finally, dynamic memory has minimal overhead to keep memory managed as opposed to getting memory under control when you have no memory available to work with.

The most important thing to understand about Dynamic memory is that you can run more virtual machines with it. Let’s say you have a 32gb of physical memory in your server; you look at task manager and you see that you are using 31gb or memory. If you have another machine that needs 2gb of memory can you start it? Answer: YES! Hyper-V will simply take “buffer” memory away from other machines to give you the memory you need to run the machine. If you need to start another 4gb server can you? Answer: Yes! As long as you have (or can get) physical memory you can start more machines. Hyper-V will grab buffer memory first, then available memory and finally it will grab real memory from lower priority machines to give you the memory you need to start your machines. If you do not have any of these (Hopefully you do not run things that tight), then the machine will not start.

Dynamic Memory Requirements?

Dynamic Memory requires the following:

 Windows Server 2012 or Hyper-V Server 2012 (click links for free downloads) {Windows 2008 R2 SP1 or above; Windows 8 or above}

 Hyper-V role.

 Hyper-V Guest with Integration Components installed

 Dynamic memory configured in the guest settings

Dynamic memory works with OLD OS’s (eg. Windows Server 2003) as well as new OS’s (Windows 8/2012) running as a guest on Hyper-V. You can find a complete compatibility list here. This incredible technology is added with the Integration components.

How To add the integration component and enable Dynamic Memory on Windows Server 2003 or other Operating Systems

1. Open the Virtual Machine Connection (Double-Click the machine in Hyper-V Manager)

2. Click the Action menu the select “Insert Integration Services Setup Disks“

3. Login to the server and run the Setup from the Integration Services Setup CD (if you get a message about uninstalling a previous version, go ahead and do it)

4. You may then have to restart the server, once you do that, the newest integration components will be available including Dynamic Memory.

How To: Enable and Configure Dynamic Memory

Enable Dynamic Memory in Hyper-V Manager under the settings for the Virtual Machine {guest}

1. Open Hyper-V Manager. {Start Keyboard Button – Hyper-V Manager}

2. Click on your Host Machine Name in the left pane {This will display your Virtual Machines; if you do not have virtual machines, you will need to create them. see the lab “Install/Enable and Configure Hyper-V Getting Started” at http://itproguru-app.azurewebsites.net/hol}

3. Right-Click Guest Machine Name {in the right pane) and choose Settings. {This will open the box shown below}

4. Click Memory – Turn on checkbox Enable Dynamic Memory then adjust settings as desired. See explanation of settings below

5. Click Apply or OK

Deep Dive into Windows Server 2012 Hyper-V Dynamic Memory – what all this stuff means/does…

1) Startup RAM: Specifies the amount of memory required to start the virtual machine. The value needs to be high enough to allow the guest operating system to start, but should be as low as possible to allow for optimal memory utilization and potentially higher consolidation ratios.

Startup RAM is the amount of physical memory the Guest MUST be given when the machine is turned on (or rebooted). If the host does not have this amount of memory it will try to get memory from dynamically changing the amount of memory on other guests on that host. If physical memory cannot be allocated due to defined limits in other guest machines then Hyper-V will not allow you to turn the machine. You should set this to the amount of memory the machine needs to efficiently boot. Depending on the workload, you may want to be stringent or aggressive. If you know you are going to be rebooting the machine in a host starving situation then be stringent and only give it what it needs to start. If you want to have plenty of memory to start quickly, then be more aggressive and give it more memory. Just remember, the amount you give it must be available to turn the machine on. In Windows Server 2012, there is a new feature that will allow you in a starving situation to reboot even if you do not have this amount of memory available. It will NOT allow you to boot a cold machine if it cannot make this amount of memory available. Startup RAM can ONLY be defined/changed while the machine is OFF.

2) Enable Dynamic Memory – turn on the checkbox to enable, turn off checkbox to disable. Cannot change setting while machine is running.

a. Minimum RAM: Minimum RAM is the low value of what the guest will be allowed to use. Hyper-V will never take more memory away from this machine than the Minimum RAM allows. This machine will ALWAYS have at least this much memory.

The required memory to run an idle virtual machine is less than the required memory to start that virtual machine. By reclaiming unused memory from the idle virtual machines, you can potentially run more virtual machines on one host. To do this, you specify a smaller value for the minimum memory than for the startup memory. However, if you do this in Windows Server 2008 R2 with SP1, a virtual machine cannot restart when the host does not have enough physical memory. This limitation is more problematic in environments where many virtual machines are idle on a host, such as pooled VDI environments and server consolidation environments that are under low load (for example, during the night). Hyper-V will auto-adjust the “assigned memory” down as the memory demand decreases or as the host needs more memory for higher priority workloads. However, it can never (regardless of priority) take more than the minimum amount of memory away from a guest. You can decrease the minimum amount of memory while the server is running but you cannot increase it (e.g. you can change the value from 1024 to 512; but you cannot change from 512 to 1024)

b. Maximum RAM: Maximum RAM is the maximum amount of memory Hyper-V will give a guest regardless of demand. This is very handy to make sure a server does not take more than their fair share. As an example you may have a development server that you want to use Dynamic memory on so you can keep it from going rogue, with a memory leak, taking more resources than you want to allow them to take.

The maximum amount will only be used if the guest actually needs it. It is an upper limit on what can be given to the virtual machine guest. Simply set the maximum and if this machine needs more, too bad. It will be as if that is all the memory that is available so the guest will have to get the memory somewhere else (like it’s page file). You can increase the maximum amount of memory while the server is running but you cannot decrease it (e.g. you can change the value from 4096 to 8192; but you cannot change from 8192 to 4096)

c. Dynamic Example in a physical world a memory upgrade requires shutting down the virtual machine, a common challenge for administrators is upgrading the maximum amount of memory for a virtual machine as demand increases.

Example 1: consider a virtual machine running IIS Server and configured with a maximum of 8 GB of RAM. Because of an increase in the size of the site or the traffic, the virtual machine now requires more memory. In Windows Server 2008 R2 with SP1, you must shut down the virtual machine to perform the upgrade, which requires planning for downtime and decreasing business productivity. With Windows Server 2012, you can apply that change while the virtual machine is running.
Example 2: Fast-growing organizations whose workloads are rapidly expanding often need to add more virtual machines to their host processors. These organizations want to optimize the number of virtual machines they can place on a host server to minimize the number of expensive host servers they need to purchase. With the Hyper-V Dynamic Memory improvements in Windows Server 2012, IT administrators can now reduce the Minimum RAM in order to load up the host with more servers without bringing anything down. This allows administrators to allocate virtual machine memory resources more efficiently and dramatically increase virtual machine consolidation ratios and uptime.

3) Memory Buffer: Specifies how much memory Hyper-V will attempt to assign to the virtual machine compared to the amount of memory actually needed by the applications and services running inside the virtual machine.

Memory buffer is specified as a percentage because the actual amount of memory that represents the buffer changes in response to changes in memory usage while the virtual machine is running. Hyper-V uses performance counters in the virtual machine that identify committed memory to determine the current memory requirements of the virtual machine and then calculates the amount of memory to add as a buffer. The buffer is determined using the following formula:

Amount of memory buffer = how much memory the virtual machine actually needs / (memory buffer value / 100).

For example, if the memory committed to the guest operating system is 1000 MB and the memory buffer is 20%, Hyper-V will attempt to allocate an additional 20% (200 MB) for a total of 1200 MB of physical memory allocated to the virtual machine.

NOTE: The buffer is not maintained when there is not enough physical memory available in the host to give every virtual machine its requested memory buffer.

NOTE: You can change the buffer while the machine is running

4) Memory Weight: Memory Weight provides Hyper-V with a way to determine how to distribute memory among virtual machines if there is not enough physical memory available in the computer to give every virtual machine its requested amount of memory.

The higher the weight, the higher the priority when it comes time to give out memory. It is also used for taking memory away. The lower the weight, the most likely that machine is to lose memory when the host needs it. Guests with a lower weight will get starved if need be in order to give higher weight guests the memory they desire.

Non-uniform memory access (NUMA)

Depending on the CPU settings of the machine, memory is managed differently. When you enable dynamic memory you also enable non-uniform memory access (NUMA).

A very import aspect of virtual machine memory configuration is Non-uniform memory access (NUMA). NUMA is a memory design that speeds up memory access by partitioning physical memory so each processor in a multi-CPU has its own memory. For example, in a system with 8 cores and 32 GB of RAM, each core or node has 4 GB of physical memory. If a virtual machine is configured to use 8 GB of RAM, the system has to use memory in another node. Because crossing the NUMA boundary can reduce virtual performance by as much as 8%, it is a best practice to configure a virtual machine to use resources from a single NUMA node.

If you only have a single processor on the virtual machine and you enable dynamic memory you may get a warning message:

“This virtual machine is configured with uniform memory access. virtual machines configured with a large amount of memory or many virtual processors can often benefit from non-uniform memory access. Disable Dynamic memory to allow non-uniform memory access.”

You can get rid of the above message by adding a second processor which allows Hyper-V to dynamically manage the memory across the processors without taking a performance hit. For more information about NUMA, refer to the following articles:

 Understanding Non-uniform Memory Access

 Determining NUMA node boundaries for modern CPUs

 NUMA Node Balancing

Memory Demand, Assigned and Status

Memory demand is how much memory your guest server needs. Memory assigned is how much memory Hyper-V has allocated to your virtual machine. Memory status is an indicator to let you know of the health state of the memory of the machine. If the status is OK then the assigned memory is higher than the demand. However, your status can go into a warning state as the demand increases above what Hyper-V has assigned. Remember, this can happen if the host is out of memory due to higher priority machines taking memory reserves away from this machine.

Hyper-V Smart Paging

Windows Server 2012 also introduces Hyper-V Smart Paging for robust virtual machine restart. Although minimum memory increases virtual machine consolidation numbers, it also brings a challenge. If a virtual machine has a smaller amount of memory than its startup memory and it is restarted, Hyper-V needs additional memory to restart the machine. Due to host memory pressure or virtual machines’ states, Hyper-V may not always have additional memory available. This can cause sporadic virtual machine restart failures in customer environments. In Windows Server 2012, Hyper-V Smart Paging is used to bridge the memory gap between minimum memory and startup memory and let virtual machines restart reliably. This gap is only temporary and should not be an issue beyond the time it takes to restart a server (10 minutes or less)

Hyper-V Smart Paging is a memory management technique that uses disk resources as additional, temporary memory when more memory is required to restart a virtual machine. This approach has both advantages and drawbacks. It provides a reliable way to keep the virtual machines running when no physical memory is available. However, it can degrade virtual machine performance because disk access speeds are much slower than memory access speeds. Ok, I said above Hyper-V DOES NOT OVERCOMMIT! Well, this is THE ONLY exception to the rule and it is only for a very limited time and reason!

To minimize the performance impact of Smart Paging, Hyper-V uses it only when all of the following occur:

 The virtual machine is being restarted.

 No physical memory is available.

 No memory can be reclaimed from other virtual machines that are running on the host.

Hyper-V Smart Paging is not used when:

 A virtual machine is being started from an off state (instead of a restart).

 Oversubscribing memory for a running virtual machine would result.

 A virtual machine is failing over in Hyper-V clusters.

What if you are just OUT of physical Memory?

There will come a time when you simply have more workloads running that you have resources to run so what happens? Hyper-V continues to rely on internal guest paging when host memory is oversubscribed because it is more effective than Hyper-V Smart Paging. With internal guest paging, the paging operation inside virtual machines is performed by Windows Memory Manager. Windows Memory Manager has more information than does the Hyper-V host about memory use within the virtual machine, which means it can provide Hyper-V with better information to use when it chooses the memory to be paged. Because of this, internal guest paging incurs less overhead to the system than Hyper-V Smart Paging.

How is Dynamic Memory different from other Virtualization Platforms like ESX? [Hyper-V Dynamic Memory vs. ESX Overcommit]

Now that you fully understand how Dynamic Memory works, let’s look at how it is different from other solutions. Specifically, let’s look at VMWare and how they manage memory. VMware ESX leverages several techniques to manage memory. They use transparent page sharing, ballooning, and host swapping. Collectively, they call these technologies “Memory Overcommit”

a. Transparent Page Sharing – multiple virtual machines that have identical sets of memory content use the SAME page of physical memory. Now this sounds good right, two machines have exactly the same thing in memory, let’s leverage that knowledge and have the machine share memory. In theory, this is great. However, in practice, you have just opened a HUGE security hole because now two different machines and sharing the same memory space. This is not allowed in high security environments and is risky in all environments. Additionally this is very expensive. In order to use this technology, the hypervisor must hash all the memory pages and then compare the hashes. If the hashes are identical, it then has to compare the actual pages to make sure they are the same. Then when a guest actually rights to that shared space the hypervisor, will page fault and the error handler for the page fault will create a local copy of the page, then allow the right to happen on its own dedicated memory page. This is incredibly expensive in terms of CPU utilization and yes, you guessed it, expensive in memory utilization while doing all these comparisons. Personally, the only thing I really like about this technology is you can adjust the settings to disable it . PS. Not trying to be mean, just stating my opinion as a security and performance aware IT Administrator.

b. Ballooning – Ballooning is very similar to the way Dynamic Memory works in Hyper-V. A “driver” is installed on the guest OS. That balloon driver then polls the hypervisor for information on how much memory it can have, how much it needs and does the work of sizing, claiming and reclaiming memory on the guest. The ballooning in ESX is different from the way Hyper-V and Integration Components driver work but to go into that in detail would be an entire blog post itself. With ballooning if the guest OS is starving and cannot get more memory from the host, the guest will page to disk. This is where the paging should happen! I am a fan!

c. Hypervisor swapping – Hypervisor swapping is when the hypervisor grabs additional memory from the host by allocating page file memory for the guest OS to use. You guessed it, this is very expensive because now the HOST is using swap/page memory (very slow disk memory). The memory that is swapped is being done by the hypervisor and is created for and on behalf of a single guest. So from a security standpoint that is good. However, the hypervisor has to do all the work which means that not only is the guest going to get a huge performance hit (and not know why because the guest does not even know it is swapping) but the hypervisor will too because it is doing all the work. There are advantages to this that include the guest does not have as much to do. However, I would rather put the burden on the guest instead of the hypervisor. Additionally, you could end up with a double paging problem which happens when the hypervisor is paging for a guest and then the guest also starts paging. Talk about performance hit. You can probably guess from my comments about Transparent Page Sharing above how I am going to feel about this one. Wow, where I liked the concept of transparent paging (but not much more) I do not even like the concept of Hypervisor swapping. This is just a bad way of overcommitting memory.

Hyper-V Dynamic Memory vs. ESX Memory Overcommit

Now that we understand dynamic memory and memory overcommit. Let’s compare them at a high level.

Keep the Conversation Going: What do you think about dynamic memory? If you have used it, what improvements in density have you seen? What else would you like to see in Dynamic Memory? What scenarios have you used it? Any scenarios where you would not use it? (Exchange, SQL, others?) Please add your comments or Tweet:

@ITProGuru Dynamic Memory-Much Different than Over Commit–Become Virtualization Expert #VirtExpert http://shar.es/jQIj9

Also of interest:

 Lab Guide – Windows 2012 RTM Install and Configure Windows and Hyper-V

 Build Your Own Windows Server 2012 Lab

 Hands On Lab Guides

 Free: Download Windows Server 2012

 Free: Download Hyper-V Server 2012