Welcome to Knowledge Base!

VMware Dead Space Overview

There are two place that dead space can be introduced:

• VMFS — When an administrator deletes a virtual disk or an entire virtual machine (or moves it to another datastore) the space that used to store that virtual disk or virtual machine is now dead on the array. The array does not know that the space has been freed up, therefore, turning those blocks into dead space.
• In-guest — When a file has been moved or deleted from a guest files system inside of a virtual machine on a virtual disk, the underlying VMFS does not know that a block is no longer in use by the virtual disk and consequently neither does the array. So that space is now also dead space.

So dead space can be accumulated in two ways. Fortunately, VMware has methods for dealing with both, that leverage the UNMAP feature support of the FlashArray.

esxcli storage vmfs unmap -l <datastore name> -n (blocks per iteration)

Unmap: Async Unmapped 5000 blocks from volume 545d6633-4e026dce-d8b2-90e2ba392174 

fsutil behavior query DisableDeleteNotify

fsutil behavior set DisableDeleteNotify 0

$wql = "SELECT Label, Blocksize, Name FROM Win32_Volume WHERE FileSystem='NTFS'"
Get-WmiObject -Query $wql -ComputerName '.' | Select-Object Label, Blocksize, Name

pureuser@ubuntu:/mnt$ sudo mount /dev/sdd /mnt/unmaptest -o discard

Space Reclamation with VMFS

Space reclamation with VMFS differs depending on the version of ESXi. VMware has supported UNMAP in various forms since ESXi 5.0. This document is only going to focus on UNMAP implementation for ESXi 5.5 and later. For previous UNMAP behaviors, refer to VMware documentation.

In vSphere 5.5 and 6.0, VMFS UNMAP is a manual process, executed on demand by an administrator. In vSphere 6.5, VMFS UNMAP is an automatic process that gets executed by ESXi as needed without administrative intervention.

esxcli storage vmfs unmap -l <datastore name> -n (blocks per iteration)

Unmap: Async Unmapped 5000 blocks from volume 545d6633-4e026dce-d8b2-90e2ba392174 

VMFS UNMAP in vSphere 5.5 through 6.0

To reclaim space in vSphere 5.5 and 6.0, UNMAP is available in the command “esxcli”. UNMAP can be run anywhere esxcli is installed and therefore does not require an SSH session:

esxcli storage vmfs unmap -l <datastore name> -n (blocks per iteration)

UNMAP with esxcli is an iterative process. The block count specifies how large each iteration is. If you do not specify a block count, 200 blocks will be the default value (each block is 1 MB, so each iteration issues UNMAP to a 200 MB section at a time). The operation runs UNMAP against the free space of the VMFS volume until the entirety of the free space has been reclaimed. If the free space is not perfectly divisible by the block count, the block count will be reduced at the final iteration to whatever amount of space is left.

While the FlashArray can handle very large values for this operation, ESXi does not support increasing the block count any larger than 1% of the free capacity of the target VMFS volume. Consequently, the best practice for block count during UNMAP is no greater than 1% of the free space. So as an example, if a VMFS volume has 1,048,576 MB free, the largest block count supported is 10,485 (always round down). If you specify a larger value the command will still be accepted, but ESXi will override the value back down to the default of 200 MB, which will dramatically slow down the operation.

It is imperative to calculate the block count value based off of the 1% of the free space only when that capacity is expressed in megabytes—since VMFS 5 blocks are 1 MB each. This will allow for simple and accurate identification of the largest allowable block count for a given datastore. Using GB or TB can lead to rounding errors, and as a result, too large of a block count value. Always round off decimals to the lowest near MB in order to calculate this number (do not round up).

There are other methods to run or even schedule UNMAP, such as PowerCLI, vRealize Orchestrator and the FlashArray vSphere Web Client Plugin. These methods are outside of the scope of this document, please refer to the respective VMware and FlashArray integration documents for further detail.

If an UNMAP process seems to be slow, you can check to see if the block count value was overridden. You can check the hostd.log file in the /var/log/ directory on the target ESXi host. For every UNMAP operation there will be a series of messages that dictate the block count for every iteration. Examine the log and look for a line that indicates the UUID of the VMFS volume being reclaimed, the line will look like the example below:

Unmap: Async Unmapped 5000 blocks from volume 545d6633-4e026dce-d8b2-90e2ba392174 

VMFS UNMAP in vSphere 6.5

In the ESXi 6.5 release, VMware introduced automatic UNMAP support for VMFS volumes. ESXi 6.5 introduced a new version of VMFS, version 6. With VMFS-6, there is a new setting for all VMFS-6 volumes called UNMAP priority. This defaults to low.

Pure Storage recommends that this be configured to “low” and not disabled. VMware only offers a low priority for ESXi 6.5—medium and high priorities were not enabled in the ESXi kernel.

Automatic UNMAP with vSphere 6.5 is an asynchronous task and reclamation will not occur immediately and will typically take 12 to 24 hours to complete. Each ESXi 6.5 host has a UNMAP “crawler” that will work in tandem to reclaim space on all VMFS-6 volumes they have access to. If, for some reason, the space needs to be reclaimed immediately, the esxcli UNMAP operation described in the previous section can be run.

Please note that VMFS-6 Automatic UNMAP will not be issued to inactive datastores. In other words, if a datastore does not have actively running virtual machines on it, the datastore will be ignored. In those cases, the simplest option to reclaim them is to run the traditional esxcli UNMAP command.

Pure Storage does support automatic UNMAP being disabled, if that is, for some reason, preferred by the customer. But to provide the most efficient and accurate environment, it is highly recommended to be left enabled.

VMFS UNMAP in vSphere 6.7 and later

In ESXi 6.7 VMware introduced a new option for utilizing automatic UNMAP as well as adding additional configuration options for the existing features available in ESXi 6.5.

The two methods available for Automatic UNMAP in ESXi 6.7 and later:

• fixed (new to ESXi 6.7)
• priority (medium and high)

Since we are already familiar with what priority based space reclamation is from the VMFS UNMAP in vSphere 6.5 section above let's start with the enhancements available in 6.7 before reviewing "fixed" space reclamation.

In ESXi 6.5 you only had one option available for automatic space reclamation (priority based) and a singular option of "low". With ESXi 6.7 that now changes and you have the added options of "medium" and "high".

The differences between the three are as follows:**

**Information on chart was found from VMware knowledge sources here.

As you will note above, priority based space reclamation process is a variable process with speed of reclamation depending on what option you have chosen. This provides for flexibility within ESXi depending on the current load of the datastore(s) on how quickly space can be recovered.

Also available in ESXi 6.7 is a new "fixed" space reclamation method. This option provides the end-user with the ability to determine how quickly (in MB/s) UNMAP operations can happen to the backing storage at a "fixed" rate. The options vary from 100 MB/s up to 2000 MB/s. Introducing this option provides the end-user with the ability to set a static rate for space reclamation as well as allowing for a much more aggressive reclamation process if the backing storage is able to ingest the higher load.

Modifying VMFS UNMAP priorities

It is while creating a new datastore when you will have the first opportunity to configure space reclamation. The options here are limited as you can only disable space reclamation (not recommended) or use priority based reclamation at low priority (default option selected).

Let's say however that you want to use the fixed space reclamation method for a higher rate of UNMAPs being sent to the underlying storage provider.

Once your datastore has been created you can right click on the datastore and select "Edit Space Reclamation". From here you can select the desired speed and save the changes.

This is illustrated below.

The last scenario is changing the priority level from "low" to "medium" or "high". As you can see from the screenshot above there are no options to modify the space reclamation priorities. As there are no options available in the GUI to modify these values the command line interface (CLI) on the ESXi host is where this change must be made. If this is something you wish to do please review the, Use the ESXCLI Command to Change the Space Reclamation Parameters, from VMware on how this change can be made.

Space Reclamation In-Guest

The discussion above speaks only about space reclamation directly on a VMFS volume which pertains to dead space accumulated by the deletion or migration of virtual machines and virtual disks. Running UNMAP on a VMFS only removes dead space in that scenario. But, as mentioned earlier, dead space can accumulate higher up in the VMware stack—inside of the virtual machine itself.

When a guest writes data to a file system on a virtual disk, the required capacity is allocated on the VMFS (if not already allocated) by expanding the file that represents the virtual disk. The data is then committed down to the array. When that data is deleted by the guest, the guest OS filesystem is cleared of the file, but this deletion is not reflected by the virtual disk allocation on the VMFS, nor the physical capacity on the array. To ensure the below layers are accurately reporting used space, in-guest UNMAP should be enabled.

fsutil behavior query DisableDeleteNotify

fsutil behavior set DisableDeleteNotify 0

$wql = "SELECT Label, Blocksize, Name FROM Win32_Volume WHERE FileSystem='NTFS'"
Get-WmiObject -Query $wql -ComputerName '.' | Select-Object Label, Blocksize, Name

pureuser@ubuntu:/mnt$ sudo mount /dev/sdd /mnt/unmaptest -o discard

Understanding In-Guest UNMAP in ESXi

Prior to ESXi 6.0 and virtual machine hardware version 11, guests could not leverage native UNMAP capabilities on a virtual disk because ESXi virtualized the SCSI layer and did not report UNMAP capability up through to the guest. So even if guest operating systems supported UNMAP natively, they could not issue UNMAP to a file system residing on a virtual disk. Consequently, reclaiming this space was a manual and tedious process.

In ESXi 6.0, VMware has resolved this problem and streamlined the reclamation process. With in-guest UNMAP support, guests running in a virtual machine using hardware version 11 can now issue UNMAP directly to virtual disks. The process is as follows:

1. A guest application or user deletes a file from a file system residing on a thin virtual disk
2. The guest automatically (or manually) issues UNMAP to the guest file system on the virtual disk
3. The virtual disk is then shrunk in accordance to the amount of space reclaimed inside of it.
4. If EnableBlockDelete is enabled, UNMAP will then be issued to the VMFS volume for the space that previously was held by the thin virtual disk. The capacity is then reclaimed on the FlashArray.

Prior to ESXi 6.0, the parameter EnableBlockDelete was a defunct option that was previously only functional in very early versions of ESXi 5.0 to enable or disable automated VMFS UNMAP. This option is now functional in ESXi 6.0 and has been re-purposed to allow in-guest UNMAP to be translated down to the VMFS and accordingly the SCSI volume. By default, EnableBlockDelete is disabled and can be enabled via the vSphere Web Client or CLI utilities.

In-guest UNMAP support does actually not require this parameter to be enabled though. Enabling this parameter allows for end-to-end UNMAP or in other words, in-guest UNMAP commands to be passed down to the VMFS layer. For this reason, enabling this option is a best practice for ESXi 6.x and later.

For more information on EnableBlockDelete and VMFS-6, you can refer to the following blog post here.

ESXi 6.5 expands support for in-guest UNMAP to additional guests types. ESXi 6.0 in-guest UNMAP only is supported with Windows Server 2012 R2 (or Windows 8) and later. ESXi 6.5 introduces support for Linux operating systems. The underlying reason for this is that ESXi 6.0 and earlier only supported SCSI version 2. Windows uses SCSI-2 UNMAP and therefore could take advantage of this feature set. Linux uses SCSI version 5 and could not. In ESXi 6.5, VMware enhanced their SCSI support to go up to SCSI-6, which allows guest like Linux to issue commands that they could not before.

Using the built-in Linux tool, sq_inq, you can see, through an excerpt of the response, the SCSI support difference between the ESXi versions:

You can note the differences in SCSI support level and also the product revision of the virtual disk themselves (version 1 to 2).

It is important to note that simply upgrading to ESXi 6.5 will not provide SCSI-6 support. The virtual hardware for the virtual machine must be upgraded to version 13 once ESXi has been upgraded. VM hardware version 13 is what provides the additional SCSI support to the guest.

The following are the requirements for in-guest UNMAP to properly function:

1. The target virtual disk must be a thin virtual disk. Thick-type virtual disks do not support UNMAP.
2. For Windows In-Guest UNMAP:
   1. ESXi 6.0 and later
   2. VM Hardware version 11 and later
3. For Linux In-Guest UNMAP:
   1. ESXi 6.5 and later
   2. VM Hardware version 13 and later
4. If Change Block Tracking (CBT) is enabled for a virtual disk, In-Guest UNMAP for that virtual disk is only supported starting with ESXi 6.5

In-Guest UNMAP Alignment Requirements

VMware ESXi requires that any UNMAP request sent down by a guest must be aligned to 1 MB. For a variety of reasons, not all UNMAP requests will be aligned as such and in in ESXi 6.5 and earlier a large percentage failed. In ESXi 6.5 Patch 1, ESXi has been altered to be more tolerant of misaligned UNMAP requests. See the VMware patch information here.

Prior to this, any UNMAP requests that were even partially misaligned would fail entirely. Leading to no reclamation. In ESXi 6.5 P1, any portion of UNMAP requests that are aligned will be accepted and passed along to the underlying array. Misaligned portions will be accepted but not passed down. Instead, the affected blocks referred to by the misaligned UNMAPs will be instead zeroed out with WRITE SAME. The benefit of this behavior on the FlashArray, is that zeroing is identical in behavior to UNMAP so all of the space will be reclaimed regardless of misalignment.

In-Guest UNMAP with Windows

Starting with ESXi 6.0, In-Guest UNMAP is supported with Windows 2012 R2 and later Windows-based operating systems. For a full report of UNMAP support with Windows, please refer to Microsoft documentation.

NTFS supports automatic UNMAP by default—this means (assuming the underlying storage supports it) Windows will issue UNMAP to the blocks a file used to consume immediately once it has been deleted or moved.

Automatic UNMAP is enabled by default in Windows. This can be verified with the following CLI command:

fsutil behavior query DisableDeleteNotify

If DisableDeleteNotify is set to 0, UNMAP is ENABLED. Setting it to 1, DISABLES it. Pure Storage recommends this value remain enabled. To change it, use the following command:

fsutil behavior set DisableDeleteNotify 0

Windows also supports manual UNMAP, which can be run on-demand or per a schedule. This is performed using the Disk Optimizer tool. Thin virtual disks can be identified in the tool as volume media types of “thin provisioned drive”—these are the volumes that support UNMAP.

Select the drive and click “Optimize”. Or configure a scheduled optimization.

$wql = "SELECT Label, Blocksize, Name FROM Win32_Volume WHERE FileSystem='NTFS'"
Get-WmiObject -Query $wql -ComputerName '.' | Select-Object Label, Blocksize, Name

Windows prior to ESXi 6.5 Patch 1

Ordinarily, this would work with the default configuration of NTFS, but VMware enforces additional UNMAP alignment, that requires a non-default NTFS configuration. In order to enable in-guest UNMAP in Windows for a given NTFS, that NTFS must be formatted using a 32 or 64K allocation unit size. This will force far more Windows UNMAP operations to be aligned with VMware requirements.

64K is also the standard recommendation for SQL Server installations—which therefore makes this a generally accepted change. To checking existing NTFS volumes are using the proper allocation unit size to support UNMAP, this simple PowerShell two-line command can be run to list a report:

$wql = "SELECT Label, Blocksize, Name FROM Win32_Volume WHERE FileSystem='NTFS'"
Get-WmiObject -Query $wql -ComputerName '.' | Select-Object Label, Blocksize, Name

Due to alignment issues, the manual UNMAP tool (Disk Optimizer) is not particularly effective as often most UNMAPs are misaligned and will fail.

Windows with ESXi 6.5 Patch 1 and Later

As of ESXi 6.5 Patch 1, all NTFS allocation unit sizes will work with in-guest UNMAP. So at this ESXi level no unit size change is required to enable this functionality. That being said, there is additional benefit to using a 32 or 64 K allocation unit. While all sizes will allow all space to be reclaimed on the FlashArray, a 32 or 64 K allocation unit will cause more UNMAP requests to be aligned and therefore more of the underlying virtual disk will be returned to the VMFS (more of it will be shrunk).

The manual tool, Disk Optimizer, now works quite well and can be used. If UNMAP is disabled in Windows (it is enabled by default) this tool can be used to reclaim space on-demand or via a schedule. If automatic UNMAP is enabled, there is generally no need to use this tool.

For more information on this, please read the following blog post here.

In-Guest UNMAP with Linux

Starting with ESXi 6.5, In-Guest UNMAP is supported with Linux-based operating systems and most common file systems (Ext4, Btrfs, JFS, XFS, F2FS, VFAT). For a full report of UNMAP support with Linux configurations, please refer to appropriate Linux distribution documentation. To enable this behavior it is necessary to use Virtual Machine Hardware Version 13 or later.

Linux supports both automatic and manual methods of UNMAP.

Linux file systems do not support automatic UNMAP by default—this behavior needs to be enabled during the mount operation of the file system. This is achieved by mounting the file system with the “discard” option.

pureuser@ubuntu:/mnt$ sudo mount /dev/sdd /mnt/unmaptest -o discard

When mounted with the discard option, Linux will issue UNMAP to the blocks a file used to consume immediately once it has been deleted or moved.

Pure Storage does not require this feature to be enabled, but generally recommends doing so to keep capacity information correct throughout the storage stack.

Linux with ESXi 6.5

In ESXi 6.5, automatic UNMAP is supported and is able to reclaim most of the identified dead space. In general, Linux aligns most UNMAP requests in automatic UNMAP and therefore is quite effective in reclaiming space.

The manual method fstrim, does align initial UNMAP requests and therefore entirely fails.

Linux with ESXi 6.5 Patch 1 and Later

In ESXi 6.5 Patch 1 and later, automatic UNMAP is even more effective, now that even the small number of misaligned UNMAPs are handled. Furthermore, the manual method via fstrim works as well. So in this ESXi version, either method is a valid option.

What to expect after UNMAP is run on the FlashArray

The behavior of space reclamation (UNMAP) on a data-reducing array such as the FlashArray is somewhat changed and this is due to the concept of data-deduplication. When a host runs UNMAP (ESXi or otherwise), an UNMAP SCSI command is issued to the storage device that indicates what logical blocks are no longer in use. Traditionally, a logical block address referred to a specific part of an underlying disk on an array. So when UNMAP was issued, the physical space was always reclaimed because there was a direct correlation between a logical block and a physical cylinder/track/block on the storage device. This is not necessarily the case on a data reduction array.

A logical block on a FlashArray volume does not refer directly to a physical location on flash. Instead, if there is data written to that block, there is just a reference to a metadata pointer. That pointer then refers to a physical location. If UNMAP is executed against that block, only the metadata pointer is guaranteed to be removed. The physical data will remain if it is deduplicated, meaning other blocks (anywhere else on the array) have metadata pointers to that data too. A physical block is only reclaimed once the last pointer on your array to that data is removed. Therefore, UNMAP only directly removes metadata pointers. The reclamation of physical capacity is only a possible consequential result of UNMAP.

Herein lies the importance of UNMAP—making sure the metadata tables of the FlashArray are accurate. This allows space to be reclaimed as soon as possible. Generally, some physical space will be immediately returned upon reclamation, as not everything is dedupable. In the end, the amount of reclaimed space heavily relies on how dedupable the data set is—the higher the dedupability, the lower the likelihood, and amount, and immediacy of physical space being reclaimed. The fact to remember is that UNMAP is important for the long-term “health” of space reporting and usage on the array.

In addition to using the Pure Storage vSphere Web Client Plugin, standard provisioning methods through the FlashArray GUI or FlashArray CLI can be utilized. This section highlights the end-to-end provisioning of storage volumes on the Pure Storage FlashArray from creation of a volume to formatting it on an ESXi host. The management simplicity is one of the guiding principles of FlashArray as just a few clicks are required to configure and provision storage to the server.