Idle Memory Tax

January 29th, 2009 by jason Leave a reply »

Memory over commit is a money/infrastructure saving feature that fits perfectly within the theme of two of virtualization’s core concepts: doing more with less hardware, and helping save the environment with greenness. While Microsoft Hyper-V offers no memory over commit or page sharing technologies, VMware has understood the value in these technologies long before VI3. I’ve mentioned this before – if you haven’t read it yet, take a look at Carl Waldspurger’s 2002 white paper on Memory Resource management in VMware ESX Server.

One of VMware’s memory over commit technologies is called Idle Memory Tax. IMT basically allows the VMKernel to reclaim unused guest VM memory by assigning a higher “cost value” to unused allocated shares. The last piece of that sentence is key – did you catch it? This mechanism is tied to shares. When do shares come into play? When there is contention for physical host RAM allocated to the VMs. Or in short, when physical RAM on the ESX host has been over committed – we’ve granted more RAM to guest VMs than we actually have on the ESX host to cover at one time. When this happens, there is contention or a battle for who actually gets the physical RAM. Share values are what determine this. I don’t want to get too far off track here as this discussion is specifically on Idle Memory Tax, but shares are the foundation so they are important to understand.

Back to Idle Memory Tax. Quite simply it’s a mechanism to take idle/unused memory from guest VMs that are hogging it in order to give that memory to another VM where it’s more badly needed. Sort of like Robin Hood for VI. By default this is performed using VMware’s balloon driver which is the more optimal of the two available methods. Out of the box, the amount of idle memory that will be reclaimed is 75% as configured by Mem.IdleTax under advanced host configuration. The VMKernel polls for idle memory in guest VMs every 60 seconds. This interval was doubled from ESX2.x where the polling period was every 30 seconds.

Here’s a working example of the scenario:

  • Two guest VMs live on an ESX/ESXi host with 8GB RAM
  • Each VM is assigned 8GB RAM and 8,192 shares. Discounting memory overhead, content based page sharing, and COS memory usage, we’ve effectively over committed our memory by 100%
  • VM1 is running IIS using only 1GB RAM
  • VM2 is running SQL and is request the use of all 8GB RAM
  • Idle Memory Tax allows the VMKernel to “borrow” 75% of the 7GB of allocated but unused RAM from VM1 and give it to VM2.  25% of the unused allocated RAM will be left for the VM as a cushion for requests for additional memory before other memory over commit technologies kick in

Here are the values under ESX host advanced configuration that we can tweak to modify the default behavior of Idle Memory Tax:

  • Mem.IdleTax – default: 75, range: 0 to 99, specifies the percent of idle memory that may be reclaimed by the tax
  • Mem.SamplePeriod – default: 60 in ESX3.x 30 in ESX2.x, range: 0 to 180, specifies the polling interval in seconds at which the VMKernel will scan for idle memory
  • Mem.IdleTaxType – default: 1 (variable), range: 0 (flat – use paging mechanism) to 1 (variable – use the balloon driver), specifies the method at which the VMKernel will reclaim idle memory. It is highly recommended to leave this at 1 to use the balloon driver as paging is more detrimental to the performance of the VM

VMware recommends that changes to Idle Memory Tax are not necessary, or even appropriate. If you get into the situation where Idle Memory Tax comes into play, you need to question the VMs that have large quantities of allocated but idle memory. Rather than allocating more memory to the VM than it needs, thus wrongly inflating its share value, consider reducing the allocated amounts of RAM to those VMs.

Advertisement

12 comments

  1. Rbrambley says:

    Balloons, Robin Hood, and taxes! Oh my! Hmm, didn’t Prince have a CD with that title? ;)

    Great post. Thanks for taking a tough concept and making it easier to understand.

  2. rafee says:

    Hi
    I am new to this VMware,I have question
    what is the difference between Balloon drive and Memory tax?
    If both services same purpose? why do we need two different concept.
    Its like reinventing a wheel?

  3. jason says:

    The Balloon Driver is a mechanism that forces memory pages in the guest to be paged out to disk. Essentially it’s a little application in the guest (installed with VMware Tools) which the ESX host will “inflate” (think intentional memory leak). Bytes in RAM get paged to disk and ESX “borrows” the memory that was made availalble, giving it to another VM.

    Idle Memory Tax is a configurable setting which leverages either the Balloon Driver (preferred) or VMkernel swap (more detrimental to the performance of the VM) to reclaim idle memory in the VM, giving it to another VM.

    What’s the difference you ask? Not a whole lot. The Balloon Driver is a memory reclamation mechanism in and of itself. Idle Memory Tax is a configurable setting which leverages the Balloon Driver memory reclamation mechanism. Also, Idle Memory Tax only borrows idle (unused) memory (think of it as a luxury tax with the luxury being excess memory in the guest not being used) whereas the Balloon Driver borrows used or unused memory in the guest at the discretion of the ESX VMkernel.

  4. Jb says:

    Hi,

    I read your article and I found it very interressting. But there is something I can’t understand. You said Idle Memory is reclaims by balloon driver (by default). But I’m wondering when this memory taxing is working (always ? only during contention ?). I was told Ballooning was only working contention but by reading you I’ve the feeling that idle memory taxing is always “running” in the background… Can you explain me this ?

    Thanks a lot

  5. jason says:

    Memory ballooning is a mechanism and the VMkernel uses it when needed to reclaim memory during periods of memory oversubscribing.

    It is my understanding that mem tax idle is ALWAYS running and just so happens to use the balloon driver as a mechanism by default to reclaim the memory because it is a proven technology/method to use.

    Jas

  6. Hi Jason,
    Following on from your last comment that ballooning is only used during oversubscribing, whereas IMT is used all the time. I actually think that perhaps this is on its head. Although the term is “idle”; it is as you said related and applied through shares. As we know shares only apply during contention. Therefore surely IMT is only used during contention.
    To me this makes more sense. TPS is used all the time (as soon as a VM is turned on, and with the large pages cavet), but ballooning and then swapping only happens during contention (via memory state percentages). And now we can add in compression just as swapping kicks in.
    What do you think? I have been going round in circles with this as many of the VMware papers use very loose terminology which tends to confuse things. They use “enabling overcommitment on the host” with “being overcommitted” as interchangeable even though they obviously aren’t.
    Forbes.

  7. OK, I’ve spent quite a bit of time digging into the depths of this one. To cut a long story short, IMT is always being calculated, but the result of the calculation is only used during contention. So when TPS isn’t cutting the mustard and a host decides it wants to reclaim memory (calculated via free memory states), then it uses IMT to figure out which VMs to recover from. It can use ballooning and swap for this. Presumably compression also fits in now.
    So IMT is really used just to figure out which VMs to reclaim from and in what proportions.
    Anyway, I’ll be writing something about this in the next couple of days. Jason, I’ll let you see a preview ;)
    If I get time, I’ll create a blog post as well.
    Cheers, Forbes.

  8. jason says:

    @Forbes Guthrie
    Almost everything I know or have written about Idle Memory Tax comes from reading VMware ESX Server Advanced Technical Design Guide by Oglesby and Herold as well as VMware Infrastructure 3 Advanced Technical Design Guide by Oglesby, Herold, and Laverick. The former (older) book actually contains more detailed information about IMT.

    Excerpt from page 60 of the VI3 ATDG:
    “Idle Memory Tax
    When memory share allocation takes effect, VMware provides a mechanism to
    prevent virtual machines from hoarding memory they may not be utilizing. Just
    because a particular server has four times the memory share priority than another
    does not mean it requires it at the time allocation takes place. VMware has
    a process that applies an idle memory tax. This associates a higher “cost value”
    to unused allocated shares than it does to memory that is actively used within a
    virtual machine. This allows the virtual machine to release it for use on other
    guests that may require it. If the virtual machine in question has a need for the
    memory, it still has the proper authority to reclaim it as it still has priority over
    the memory space. A default value of 75% of idle memory may be reclaimed by
    the tax.”

  9. Hi Jason, I came to your blog while searching in google for tax related blogs. I got interested with your title “Idle Memory Tax”, after reading the contents only I came to know that it is an technology related blog. However I got interested in your topic of discussion and read the full post. You are maintaining a very good interactive blog. All the best.

Leave a Reply