What we found was that the only systems that were affected were those writing to the disk. Fortunately our environment was well designed and very balanced so the only machines that were directly affected were those with snapshots running (and in this instance just the one server).

One side effect to note is that the only way to recover from this is to make some space so that you can apply the snapshots. This is where storage vmotion might come in handy allowing you to move a VM to a different LUN. We chose the cold move approach since I was not confident that sVmotion wouldn’t require some disk writing. When this happened to us we had over 50GB worth of snapshots to be applied (main production file share running Volume Shadow Copy for those wondering how it got so big). Took about 2.5 hours to recover from this event but at least it was at the end of the day. This is one thing to be wary about with having a large “comfort cushion” on your VMFS datastore.

Needless to say I check my VMs (manually now for the highest disk I/O servers) to ensure that backups did not leave us a little friend waiting like a scorpion in a shoe.

I plan to write a full entry in the coming week(s) to my slowly developing blog.

VMFS will let you get ‘close’ to 0KB. When you do, “Bad Things Happen” – e.g. Chuck Norris divides by zero, and the like. Troubleshooting it is an interesting thing, and usually involves some small amount of pain deleting snapshots from the CLI.

-Cody
http://professionalvmware.com

Not sure on the 3rd party tools, but with the VI-Toolkit for Windows finding snapshots is really easy.

Go here to get started:
http://vmware.com/go/powershell/

Then to list all snapshots:
connect-viserver -name (your esx or VC)
get-vm | get-snapshot

To list snapshots older than 7 days:
connect-viserver -name (your esx or VC)
get-vm | get-snapshotwhere | where { $_.Created -le (Get-Date).AddDays(-7)}

-Cody

Thanks for the write up, by happen stance, do you know if any of the third party tools will happen to support ESXi, Foundation / Enterprise in the future?

We are currently contemplating ESX vs ESXi for our Data Center.

Thanks.

Roger L

I totally agree. I’m going to create this situation in my lab to learn more about the impact and timing. Thanks for the quick response – your community contributions are much appreciated!

Tristan

I’m a little confused about a statement you make in this post. With regards to the situation where the VMFS volume fills up b/c of one or more growing snapshots – my understanding was that only VMs on the same volume that are operating with snapshots would be impacted. VMs not running snapshots are not impacted b/c they are contained within their swap and VMDK files.

I personally use HP SIM to watch my logical volume free space and I run daily schedule snaphunter jobs. Still, I want to better understand the risks.

Thanks much for the clarification.

Tristan