Welcome to Knowledge Base!

Problem

This guide shows how to re-scan and examine FC LUNS on Solaris 10/11. This is required for detecting new LUNs and to assist with basic storage connectivity troubleshooting.

Impact

Re-scanning FC LUNs on Solaris is generally non-disruptive in a well configured environment with normal load.

Solution

root@Unixarena-SOL11:~# luxadm -e port |grep CONNECTED
/devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl CONNECTED
/devices/pci@1d,700000/SUNW,qlc@3/fp@0,0:devctl CONNECTED

Unix@sol# cfgadm -al -o show_FCP_dev |grep fc-fabric
c2                             fc-fabric    connected    configured   unknown
c4                             fc-fabric    connected    configured   unknown
Unix@sol#

Unix@sol# fcinfo hba-port |grep Port
HBA Port WWN: 10000000c884bb48
HBA Port WWN: 10000000c884bb49
HBA Port WWN: 10000000c884b85c
HBA Port WWN: 10000000c884b85d
Unix@sol#

Unix@sol# luxadm -e dump_map /dev/cfg/c4
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    29900   0        50080e8008cfb814 50080e8008cfb814 0x0  (Disk device)
1    27400   0        10000000c884b85c 20000000c884b85c 0x1f (Unknown Type,Host Bus Adapter)
Unix@sol#

Unix@sol# cfgadm -al -o show_FCP_dev c2 c4

Unix@sol# cfgadm -c configure c2::50080e8008cfb814
Unix@sol# cfgadm -c configure c4::50080e8008cfb814

root@Unixarena-SOL11:~# luxadm -e port |grep CONNECTED
/devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl CONNECTED
/devices/pci@1d,700000/SUNW,qlc@3/fp@0,0:devctl CONNECTED

root@Unixarena-SOL11:~# luxadm -e forcelip /devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl

Unix@sol# cfgadm -al -o show_FCP_dev |grep fc-fabric
c2                             fc-fabric    connected    configured   unknown
c4                             fc-fabric    connected    configured   unknown
Unix@sol#
Unix@sol# luxadm -e forcelip /dev/cfg/c2

root@Unixarena-SOL11:~# luxadm display /dev/rdsk/c1txxxxxxd0s2

Understanding How FC is Configured on a Solaris Host

The following commands are useful for general SAN stack fact-finding on Solaris:

1. List the connected HBA’s:

root@Unixarena-SOL11:~# luxadm -e port |grep CONNECTED
/devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl CONNECTED
/devices/pci@1d,700000/SUNW,qlc@3/fp@0,0:devctl CONNECTED

2. Verify FC ports are connected and configured:

Unix@sol# cfgadm -al -o show_FCP_dev |grep fc-fabric
c2                             fc-fabric    connected    configured   unknown
c4                             fc-fabric    connected    configured   unknown
Unix@sol#

3. To find the HBA’s World Wide Node number (WWN), use the fcinfo command:

Unix@sol# fcinfo hba-port |grep Port
HBA Port WWN: 10000000c884bb48
HBA Port WWN: 10000000c884bb49
HBA Port WWN: 10000000c884b85c
HBA Port WWN: 10000000c884b85d
Unix@sol#

4. You can also find the WWN using luxadm command, if HBA is already connected to the FC switch.

Unix@sol# luxadm -e dump_map /dev/cfg/c4
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    29900   0        50080e8008cfb814 50080e8008cfb814 0x0  (Disk device)
1    27400   0        10000000c884b85c 20000000c884b85c 0x1f (Unknown Type,Host Bus Adapter)
Unix@sol#

From the above output, the last line shows the HBA information. In the same way you can find the other controller information as well.

5. Zoning can be verified using the below command:

Unix@sol# cfgadm -al -o show_FCP_dev c2 c4

6. If you see any controller port WWN shown as “unconfigured," then  you can initiate FC session using below mentioned command:

Unix@sol# cfgadm -c configure c2::50080e8008cfb814
Unix@sol# cfgadm -c configure c4::50080e8008cfb814

Scanning for New FC Devices and Getting LUN Information:

Note : The command luxadm probe can also be used to scan FC LUNs.

Resetting the HBA

If you are still unable to see the new LUN/DISK, then, if you have multipathing enabled, you can try to reset the HBA.  (Do not try this in critical servers unless you are confident that multipathing is correctly configured)

1. List the connected HBA.

root@Unixarena-SOL11:~# luxadm -e port |grep CONNECTED
/devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl CONNECTED
/devices/pci@1d,700000/SUNW,qlc@3/fp@0,0:devctl CONNECTED

2. Reset the HBA using forcelip option:

root@Unixarena-SOL11:~# luxadm -e forcelip /devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl

The forcelip command can be issued to the controller names as well:

Unix@sol# cfgadm -al -o show_FCP_dev |grep fc-fabric
c2                             fc-fabric    connected    configured   unknown
c4                             fc-fabric    connected    configured   unknown
Unix@sol#
Unix@sol# luxadm -e forcelip /dev/cfg/c2

3. Verify the controller status using cfgadm -al . Make sure disks didn't loose any SAN paths after the HBA reset. If everything seems to be working, then issue the forcelip command to the other controller as well.

If you are still not able to see the new FC/SAN LUNS, then, as a last resort, reboot the server and try again. Once you see the new LUN, make sure that you also see all FC paths. A minimum two FC paths is required for SAN disks.

4. To verify the FC LUN details and multipathing, enter the following command:

root@Unixarena-SOL11:~# luxadm display /dev/rdsk/c1txxxxxxd0s2

Requirements to Install vSphere Plugin 2.5.1+ from the PureStorage GUI

For instructions on installing and updating the plugin from the Pure GUI see VMware vSphere Plugin Install (Version 3.0 | Version 2.5 or Older).

• Java 1.8 (for TLS 1.1 or 1.2). If Java 1.8+ is not present DO NOT REQUEST TO UPDATE IT.
• Network access between Array and vSphere server through TCP ports 443.
• vCenter administrator privileges and PureStorage system administrator privileges, or an administrator with these privileges is available to do the installation. Administrative username and password for vCenter server.
• vSphere Web Client.

Requirements to Install and Configure vSphere Plugin Using the PluginServer Method

• Linux OS or Windows OS (either can be VM’s).
• Network access to vSphere server port 8080 or 8081.
• Administrative username and password for vCenter server.
• Windows OS does not need java 1.8 (the pluginserver has a java run-time included).
• Linux OS needs java jdk 1.8+ and jre 8+(for TLS 1.1 or 1.2).
• The pluginserver files in archive.
• vSphere Web Client
• PureStorage system administrator privileges to connect to PureStorage array within vSphere.
• TCP Ports 443 (for vCenter to send API commands to the array).
• vSphere 5.5, vsphere 6.0 and vSphere 6.5. Note: The space reclamation feature requires vSphere 6.0 or higher and ESXi 6.0 or higher.

Troubleshooting

If after following the documentation for VMware vSphere Plugin Installation and the plugin still does not function properly, this is often due to the VMware vSphere web client server not supporting TLS 1.1 or 1.2. See Analyzing the vsphere_client_virgo.log below if you would like to confirm this.

Installing the PluginServer

The procedure below does a few different things. It runs the unregisterplugin sh/bat using the IP address of the vSphere server. It commands this server to stop its attempt to download the vSphere plugin from our array through the API and then attempts installation of the plugin. The next time a user logs into the vSphere web client, the web client triggers the plugin's installation when a user logs in.  If there was a failed attempt to prevent login from taking a long time, it will not try and install the plugin twice. Restarting the vSphere Web Client service resolves this. Next, the startserver enables the API to pull the needed files from the Windows or Linux OS. And finally, the registerplugin uses API to pull the needed files to the vSphere server and tell vSphere to install the plugin the next time someone logs into the vSphere web client.

[2016-02-11 16:06:03.213] ERROR [ERROR] http-bio-9443-exec-16         com.purestorage.FlashArrayHelper      javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair

[2017-06-19 10:34:00.954] [ERROR] vc-service-pool-2169   70002699 100142 200004 com.vmware.vise.vim.extension.VcExtensionManager    Error unzipping https://192.168.41.131/download/pure...?version=2.5.1 to directory C:\ProgramData\VMware\vSphere Web Client\vc-packages\vsphere-client-serenity\com.purestorage.plugin.vsphere-2.5.1, check if the server process has Write Permission on this machine. java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(Unknown Source)
        at java.net.SocketInputStream.read(Unknown Source)
        at sun.security.ssl.InputRecord.readFully(Unknown Source)
        at sun.security.ssl.InputRecord.read(Unknown Source)

Windows

If any of these steps do not work see the notes section below.

1. Download PureStorage_vSphere_installer.jar and the PluginServer-2.5.1_201704051804+163c8cd-rel_2_5_x.zip or latest plugin server from https://archive.dev.purestorage.com/flasharray/purity/customer_shipped_releases/older_releases_at_.._.._.._customer_shipped_releases/vsphere_plugin/ and then copy these files to a google drive (or equivalent) to share.
2. Make a new directory for the PluginServer files and unzip them to the directory.
   1. Enter make a folder in the gui called pluginserver or in CLI mkdir [drive:]path/pluginserver .
   2. Extract Pluginserver-2.5.1_201704051804+163c8cd-rel_2_5_x.zip or latest to the pluginserver folder.
3. Extract the PluginServer to above folder and make sure PureStorage_vSphere_installer.jar is present in the same folder as the unzipped PluginServer files.
   1. Enter cd pluginserver/ .
   2. Enter copy ../PureStorage_vSphere_installer.jar /pluginserver .
4. Unregister the failed plugin install.
   1. Enter unregisterplugin.bat , while in the folder where the extracted files and the PureStorage_vSphere_installer.jar resides.
   2. Enter the vSphere server'ss IP address.
   3. Enter the vSphere credentials.
5. Run startserver.bat in the command console used for creating the directory. This will start up the web server that hosts the plugin. Keep this console open and running while you perform the rest of the steps.
   1. You should see something similar to the following:

      robm$ ./startserver.bat
      Found these arguments port(8080) keystore(keystore.jks)
      File location(purestorage-vsphere-plugin.zip)
      Starting server on port 8080...
      Server started successfully!

   2. Leave this running and open a new command line window for the next step.
   3. If it doesn’t work and you see “Found these arguments port(8080)” … “Address already in use” follow the steps in the note section below.
6. In a new command window, run registerplugin.bat while in the directory where unzip extracted the pluginserver and the PureStorage_vSphere_installer.jar resides.
   1. Once run, it will ask for the IP address and credentials to run API commands against the vSphere server.
7. Restart the vSphere Web Client service on the vCenter server and then wait five minutes for it to fully come up( on vCenter 6.5, you won't see the vSphere Web Client service listed in Services, so you can use a command prompt to stop and start the process:)

C:\Program Files\VMware\vCenter Server\bin>service-control --stop vspherewebclientsvc
C:\Program Files\VMware\vCenter Server\bin>service-control --start vspherewebclientsvc

Now have the user log into the vSphere web client to see the PureStorage plugin, if it is there, you can now close both command line windows.

8. If we still do not see the PureStorage Plugin make sure they have logged out then back into the vSphere Web Client.

9. If we still do not see PureStorage Plugin proceed to Analyzing the vsphere_client_virgo.log (steps below).

Once the user has logged into vSphere Web Client and can see the plugin, follow the vSphere Web Client user guide.

Linux VCSA Appliance, Installing the Plugin Server on a Linux VM

vCenter Server Appliance VCSA requires another OS to run the pluginserver scripts on. This can be another temporary VM running Linux or Windows, the following instructions are for a linux VM. Follow the above Windows steps if this other OS is running windows OS.

If any of these steps do not work see the notes section below.

1. vCenter Server Appliance VCSA requires another machine to run the pluginserver scripts on. This can be another temporary VM running Linux or Windows (above windows steps) or can be the Windows Guest OS the customer is using for their machine to temporarily run the pluginserver scripts.
2. Download PureStorage_vSphere_installer.jar from and the PluginServer-2.5.1_201704051804+163c8cd-rel_2_5_x.zip or latest plugin server from https://archive.dev.purestorage.com/flasharray/purity/customer_shipped_releases/older_releases_at_.._.._.._customer_shipped_releases/vsphere_plugin/.
3. From the command line make new directory for the pluginserver files and unzip them to the directory.
   1. Enter mkdir pluginserver .
   2. Enter unzip -d pluginserver/ Pluginserver-2.5.1_201704051804+163c8cd-rel_2_5_x.zip .
4. Extract the PluginServer to any directory and make sure to copy PureStorage_vSphere_installer.jar to the same directory as the unzipped pluginserver files.
   1. Enter cd pluginserver/ .
   2. Enter cp ../PureStorage_vSphere_installer.jar ./ .
5. Make the scripts executable.
   1. Enter chmod a+x *.sh .
6. Unregister the failed plugin install.

   1. Enter ./unregisterplugin.bat, while in the directory where unzip extracted the files and the PureStorage_vSphere_installer.jar resides.

   2. Enter the vSphere servers ip address.
   3. Enter the vSphere credentials.
7. Execute the plugin using the pluginserver.
   1. ./startserver.sh .
   2. You should see something similar to the following:

      robm$ ./startserver.sh
      Found these arguments port(8080) keystore(keystore.jks)
      File location(purestorage-vsphere-plugin.zip)
      Starting server on port 8080...
      Server started successfully!

   3. If it doesn’t work and you see “Found these arguments port(8080)” … “Address already in use” follow the steps in the note section below.
   4. If this fails for another reason, you may need to java jdk 1.8+ and jre 8+ on this linux VM. If this is a production VM, you do not want to update java, use a nonproduction VM.
   5. In testing, a ubuntu-16.04.3 VM was used. The following commands were required:.

      robm@ubuntu:~$ sudo apt-get update
      robm@ubuntu:~$ sudo apt-get install default-jre

8. Leave this running and open a new command line window for the next step.
9. In a new command window, run registerplugin.sh while in the directory where unzip extracted the PluginServer and where the PureStorage_vSphere_installer.jar resides.
   1. Once run, the program prompts for the IP address and credentials to run API commands against the vSphere server.
10. Restart the vSphere Web Client service from the command line on the vCenter server (not the VM) and then wait five minutes for it to fully come up.
    1. Enter service vsphere-client restart .
11. Now have the user log into the vSphere web client to see the PureStorage plugin, if it is there, you can now close both command line windows.
    1. If you still do not see the PureStorage Plugin, make sure you have logged out then back into the vSphere Web Client.
    2. If you still do not see PureStorage Plugin, proceed to Analyzing the vsphere_client_virgo.log (steps below).
12. Once the user has logged into vSphere Web Client and can see the plugin, follow the vSphere Web Client user guide.

Analyzing the vsphere_client_virgo.log

Locate the vsphere_client_virgo.log and have the customer copy them to a text file then email them to us. These logs often get zipped up and numbered so be sure we have the file for the time period of when we attempted to install the plugin. Below is where these files should be located. Often this file is not where we expect it to be so searching it may be necessary. The following is from this vmware kb.

1. For vSphere 5.0, all the logs for the vSphere Web Client service are located at:
   • Windows: C:\Program Files\VMware\Infrastructure\vSphere Web Client\DMServer\serviceability\.
   • Linux: /usr/lib/vmware-vsphere-client/server/serviceability/.
2. For vSphere 5.1, all the logs for the vSphere Web Client service are located at:
   • Windows: C:\ProgramData\VMware\vSphere Web Client\serviceability\.
   • Linux: /var/log/vmware/vsphere-client/serviceability/.
3. For vSphere 5.5, all the logs for the vSphere Web Client service are located at:
   • Windows: C:\ProgramData\VMware\vSphere Web Client\serviceability\.
   • Linux: /var/log/vmware/vsphere-client/.
4. For vSphere 6.0+ as per the following VMware kb.
   • C:\ProgramData\VMware\vCenterServer\logs\vsphere-client\logs.
   • /var/log/vmware/vsphere-client/logs.

Search the vsphere_client_virgo log for when the plugin was attempted. Usually, it will say “purestorage” when the attempt to install was performed. The following are errors we have seen in this file.

[2016-02-11 16:06:03.213] ERROR [ERROR] http-bio-9443-exec-16         com.purestorage.FlashArrayHelper      javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair

This alert was from not having JDK 1.8 see this JIRA.

[2017-06-19 10:34:00.954] [ERROR] vc-service-pool-2169   70002699 100142 200004 com.vmware.vise.vim.extension.VcExtensionManager    Error unzipping https://192.168.41.131/download/pure...?version=2.5.1 to directory C:\ProgramData\VMware\vSphere Web Client\vc-packages\vsphere-client-serenity\com.purestorage.plugin.vsphere-2.5.1, check if the server process has Write Permission on this machine. java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(Unknown Source)
        at java.net.SocketInputStream.read(Unknown Source)
        at sun.security.ssl.InputRecord.readFully(Unknown Source)
        at sun.security.ssl.InputRecord.read(Unknown Source)

This alert is due to vSphere trying to establish communication with the array using TLSv1.0 ES-27873.

If installing from the GUI and it does not work and you see the above messages, install the PluginServer. If the vSphere server resides on a windows OS the PluginServer can be installed on it as described above. If the Linux VCSA appliance is being used we can put the PluginServer on a windows OS server (vm or bare metal) or a Linux vm as described above. Make sure this is not a production Linux VM as we may need to update or install java to 1.8+.

Notes:

If running the startserver.bat/sh script fails, there may be an issue when port 8080 is already in use.  As seen below:

robm$Found these arguments port(8080) keystore(deystore.jks)
File location(purestorage-vsphere-plugin.zip)
java.net.BindException: Address already in use: bind”
robm$ ./startserver.sh
Found these arguments port(8080) keystore(keystore.jks)
      File location(purestorage-vsphere-plugin.zip)
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:433)
    at sun.nio.ch.Net.bind(Net.java:425)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at sun.net.httpserver.ServerImpl.<init>(ServerImpl.java:100)
    at sun.net.httpserver.HttpsServerImpl.<init>(HttpsServerImpl.java:50)
    at sun.net.httpserver.DefaultHttpServerProvider.createHttpsServer(DefaultHttpServerProvider.java:39)
    at com.sun.net.httpserver.HttpsServer.create(HttpsServer.java:90)
    at com.purestorage.PluginServer.main(Unknown Source)
Exception in thread "main" java.lang.NullPointerException: null SSLContext
    at com.sun.net.httpserver.HttpsConfigurator.<init>(HttpsConfigurator.java:82)
    at com.purestorage.PluginServer$1.<init>(Unknown Source)
    at com.purestorage.PluginServer.main(Unknown Source)
 

To fix this error modify the startserver.bat/sh so it is using 8081.

1. Open startserver.bat/sh in a text editor (notepad or vim).
2. Search for 8080 then change it to 8081 twice.
3. Change port 8080 to 8081 in the registerplugin.bat/sh.

Here is what the startserver.bat/sh looks like after modification:

#!/bin/bash
set -e
 
if [ ! -f /usr/bin/java ]; then
   echo 'Please install java'
   exit -1
fi
 
# The listening port is 8081.  You can modify to use any port.  Make sure to modify the registerPlugin script.
java -cp ./PureStorage_PluginServer.jar com.purestorage.PluginServer 8081 keystore.jks purestorage-vsphere-plugin.zip

Additionally, you will need to edit the registerplugin.bat/sh with the following command:

java -cp ./PureStorage_PluginServer.jar:./PureStorage_vSphere_installer.jar com.purestorage.RegisterPlugin 8081 3.0.0 $ip

Additional troubleshooting steps include:

1. If you get a JNI error when running the registerserver script, most likely the installer.jar isn't in the same directory as the unzipped PluginServer scripts.
2. To determine where the plugin is failing, you can verify if it is showing up in the https://vcenterIP/mob.
   1. Go to https://ipaddress_of_vSphere_server/mob.
   2. Go to Content > ExtensionManager > extensionList["com.purestorage.plugin.vsphere"] > client .
   3. From there verify the URL and ensure that this is the correct location vCenter should be looking to download and install the plugin.
3. You can also ensure that the vCenter GUI has the plugin enabled via the following:
   1. Go to https://ipaddress_of_vSphere_server/.
   2. Go to Home > Administration > Client Plug-Ins .
   3. From there make sure the Pure Storage Plugin is enabled (and not set to disabled).

Problem

When creating a VVol datastore on a Cisco UCS Server with Fibre Channel, failures can occur when an older driver is in use. The older fnic driver cannot detect protocol endpoints as it does not support sub-luns (VVols).

Impact

The FlashArray vSphere Plugin fails to mount the VVol datastore with the error:

The following hosts do not have a valid protocol endpoint connection to the selected Pure Storage Array

Or when mounting manually, the datastore is marked on the host as inaccessible.

The /var/log/vmkernel.log file on the ESXi host will show the following VVol PE warnings when the “Rescan Storage” is initiated:

2018-01-09T18:04:42.098Z cpu5:65799)WARNING: ScsiPath: 705: Sanity check failed for path vmhba0:C0:T1:L1. The path to a VVol PE comes from adapter vmhba0 which is not PE capable. Path dropped.

The problem is likely caused by outdated scsi-fnic Cisco UCS drivers.

To check for general support for sub-luns (VVols), run the following command:

esxcli storage core adapter list

Look for Second Level Lun ID in the Capabilities column.

Solution

esxcli software vib get -n scsi-fnic

esxcli software vib install -v <full_path_to driver_file>

esxcli software vib install -v /tmp/scsi-fnic_1.6.0.37-1OEM.600.0.0.2494585.vib

Check Installed Version Of scsi-fnic Cisco UCS Driver

1. Log in to ESXi host and execute

esxcli software vib get -n scsi-fnic

Update scsi-fnic Cisco UCS Driver

To install the new driver version:

1. Download the updated driver package from http://software.cisco.com

or at

https://my.vmware.com/group/vmware/details?productId=491&downloadGroup=DT-ESX60-CISCO-FNIC-16033

2. Copy scsi-fnic vib file to the host which needs updating
3. As root execute the following command on the ESXi host:

esxcli software vib install -v <full_path_to driver_file>

Example:

esxcli software vib install -v /tmp/scsi-fnic_1.6.0.37-1OEM.600.0.0.2494585.vib

The installation result should look similar to the output below:

Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: CSCO_bootbank_scsi-fnic_1.6.0.37-1OEM.600.0.0.2494585
VIBs Removed: CSCO_bootbank_scsi-fnic_1.6.0.36-1OEM.600.0.0.2494585
VIBs Skipped:

Volume Sizing and Count

A common question when first provisioning storage on the FlashArray is what capacity should I be using for each volume? VMware VMFS supports up to a maximum size of 64 TB. The FlashArray supports far larger than that, but for ESXi, volumes should not be made larger than 64 TB due to the filesystem limit of VMFS.

Using a smaller number of large volumes is generally a better idea today. In the past a recommendation to use a larger number of smaller volumes was made for performance limitations that no longer exist.

This limit traditionally was due to two reasons :

• VMFS scalability issues due to locking
• Per-volume queue limitations on the underlying array.

VMware resolved the first issue with the introduction of Atomic Test and Set (ATS), also called Hardware Assisted Locking.

Prior to the introduction of VAAI ATS, VMFS used LUN-level locking via full SCSI-2 reservations to acquire exclusive metadata control for a VMFS volume. In a cluster with multiple nodes, all metadata operations were serialized and hosts had to wait until whichever host, currently holding a lock, released that lock. This behavior not only caused metadata lock queues but also prevented standard I/O to a volume from VMs on other ESXi hosts which were not currently holding the lock.

With VAAI ATS, the lock granularity is reduced to a much smaller level of control (specific metadata segments, not an entire volume) for the VMFS that a given host needs to access. This behavior makes the metadata change process not only very efficient, but more importantly provides a mechanism for parallel metadata access while still maintaining data integrity and availability. ATS allows for ESXi hosts to no longer queue metadata change requests, which consequently speeds up operations that previously had to wait for a lock. Therefore, situations with large amounts of simultaneous virtual machine provisioning operations will see the most benefit.

The standard use cases benefiting the most from ATS include:

• High virtual machine to VMFS density.
• Extremely dynamic environments—numerous provisioning and de-provisioning of VMs (e.g. VDI using non-persistent linked-clones).
• High intensity virtual machine operations such as boot storms, or virtual disk growth.

The introduction of ATS removed scaling limits via the removal of lock contention; thus, moving the bottleneck down to the storage, where many traditional arrays had per-volume I/O queue limits. This limited what a single volume could do from a performance perspective as compared to what the array could do in aggregate. This is not the case with the FlashArray.

A FlashArray volume is not limited by an artificial performance limit or an individual queue. A single FlashArray volume can offer the full performance of an entire FlashArray, so provisioning ten volumes instead of one, is not going to empty the HBAs out any faster. From a FlashArray perspective, there is no immediate performance benefit to using more than one volume for your virtual machines.

The main point is that there is always a bottleneck somewhere, and when you fix that bottleneck, it is transferred somewhere in the storage stack. ESXi was once the bottleneck due to its locking mechanism, then it fixed that with ATS. This, in turn, moved the bottleneck down to the array volume queue depth limit. The FlashArray doesn’t have a volume queue depth limit, so now that bottleneck has been moved back to ESXi and its internal queues.

Altering VMware queue limits is not generally needed with the exception of extraordinarily intense workloads. For high-performance configuration, refer to the section of this document on ESXi queue configuration.

VMFS Version Recommendations

Pure Storage recommends using the latest supported version of VMFS that is permitted by your ESXi host.

For ESXi 5.x through 6.0, use VMFS-5. For ESXi 6.5 and later it is highly recommended to use VMFS-6. It should be noted that VMFS-6 is not the default option for ESXi 6.5, so be careful to choose the correct version when creating new VMFS datastores in ESXi 6.5.

Example of VMFS-6  in ESXi 6.5:

When upgrading to ESXi 6.5, there is no in-place upgrade path from VMFS-5 to VMFS-6. Therefore, it is recommended to create a new volume entirely, format it as VMFS-6, and then Storage vMotion all virtual machines from the old VMFS-5 datastore to the new VMFS-6 datastore. Once the migration is completed you can then delete and remove the VMFS-5 datastore from the ESXi host and FlashArray.

Datastore Performance Management

ESXi and vCenter offer a variety of features to control the performance capabilities of a given datastore. This section will overview FlashArray support and recommendations for these features.

esxcli storage core device list –d <naa.xxxxx>
Device Max Queue Depth: 96
No of outstanding IOs with competing worlds: 64

Queue Depth Limits

ESXi offers the ability to configure queue depth limits for devices on an HBA or iSCSI initiator. This dictates how many I/Os can be outstanding to a given device before I/Os start queuing in the ESXi kernel. If the queue depth limit is set too low, IOPS and throughput can be limited and latency can increase due to queuing. If too high, virtual machine I/O fairness can be affected and high-volume workloads can affect other workloads from other virtual machines or other hosts. The device queue depth limit is set on the initiator and the value (and setting name) varies depending on the model and type:

Changing these settings require a host reboot. For instructions to check and set these values, please refer to this VMware KB article:

Changing the queue depth for QLogic, Emulex, and Brocade HBAs

Disk Schedule Number Requests Outstanding (DSNRO)

T here is a second per-device setting called “Disk Schedule Number Requests Outstanding” often referred to as DSNRO. This is a hypervisor-level queue depth limit that provides a mechanism for managing the queue depth limit for an individual device. This value is a per-device setting that defaults to 32 and can be increased to a value of 256.

It should be noted that this value only comes into play for a volume when that volume is being accessed by two or more virtual machines on that host. If there is more than one virtual machine active on it, the lowest of the two values (DSNRO or the HBA device queue depth limit) is the value that is observed by ESXi as the actual device queue depth limit. So, in other words, if a volume has two VMs on it, and DSNRO is set to 32 and the HBA device queue depth limit is set to 64, the actual queue depth limit for that device is 32. For more information on DSNRO see the VMware KB here:

Setting the Maximum Outstanding Disk Requests for virtual machines

In general, Pure Storage does not recommend changing these values. The majority of workloads are distributed across hosts and/or not intense enough to overwhelm the default queue depths. The FlashArray is fast enough (low enough latency) that the workload has to be quite high in order to overwhelm the queue.

If the default queue depth is consistently overwhelmed, the simplest option is to provision a new datastore and distribute some virtual machines to the new datastore. If a workload from a single virtual machine is too great for the default queue depth, then increasing the queue depth limit is the better option.

If a workload demands queue depths to be increased, Pure Storage recommends making both the HBA device queue depth limit and DSNRO equal.

You can verify the values of both of these for a given device with the command:

esxcli storage core device list –d <naa.xxxxx>
Device Max Queue Depth: 96
No of outstanding IOs with competing worlds: 64

Dynamic Queue Throttling

ESXi supports the ability to dynamically throttle a device queue depth limit when an array volume has been overwhelmed. An array volume is overwhelmed when the array responds to an I/O request with a sense code of QUEUE FULL or BUSY. When a certain number of these are received, ESXi will reduce the queue depth limit for that device and slowly increase it as conditions improve.

This is controlled via two settings:

• Disk.QFullSampleSize —The count of QUEUE FULL or BUSY conditions it takes before ESXi will start throttling. Default is zero (feature disabled)
• Disk.QFullThreshold —The count of good condition responses after a QUEUE FULL or BUSY required before ESXi starts increasing the queue depth limit again

The Pure Storage FlashArray does not advertise a QUEUE FULL condition for a volume. Since every volume can use the full performance and queue of the FlashArray, this limit is unrealistically high and this sense code will likely never be issued. Therefore, there is no reason to set or alter these values for Pure Storage FlashArray volumes because QUEUE FULL should rarely (or never) occur.

Storage I/O Control

VMware vCenter offers a feature called Storage I/O Control (SIOC) that will throttle selected virtual machines when a certain average latency has been reached or when a certain percentage of peak throughput has been hit on a given datastore. ESXi throttles virtual machines by artificially reducing the number of slots that are available to it in the device queue depth limit.

Pure Storage fully supports enabling this technology on datastores residing on the FlashArray. That being said, it may not be particularly useful for a few reasons.

First, the minimum latency that can be configured for SIOC before it will begin throttling a virtual machine is 5 ms.

When a latency threshold is entered, vCenter will aggregate a weighted average of all disk latencies seen by all hosts that see that particular datastore. This number does not include host-side queuing, it is only the time it takes for the I/O to be sent from the SAN to the array and acknowledged back.

Furthermore, SIOC uses a random-read injector to identify the capabilities of a datastore from a performance perspective. At a high-level, it runs a quick series of tests with increasing numbers of outstanding I/Os to identify the throughput maximums via high latency identification. This allows ESXi to determine what the peak throughput is, for when the “Percentage of peak throughput” is chosen.

Knowing these factors, we can make these points about SIOC and the FlashArray:

1. SIOC is not going to be particularly helpful if there is host-side queuing since it does not take host-induced latency into account. This (the ESXi device queue) is generally where most of the latency is introduced in a FlashArray environment.
2. The FlashArray will rarely have sustained latency above 1 ms, thus reducing the likely hood that this threshold will be reached for any meaningful amount of time on a FlashArray volume.
3. A single FlashArray volume does not have a queue limit, so it can handle quite a high number of outstanding I/O and throughput (especially reads), therefore SIOC and its random-read injector cannot identify FlashArray limits in meaningful ways.

In short, SIOC is fully supported by Pure Storage, but Pure Storage makes no specific recommendations for configuration.

Storage DRS

VMware vCenter also offers a feature called Storage Dynamic Resource Scheduler (Storage DRS / SDRS). SDRS moves virtual machines from one datastore to another when a certain average latency threshold has been reach on the datastore or when a certain used capacity has been reached. For this section, let’s focus on the performance-based moves.

Storage DRS, like Storage IO Control, waits for a certain latency threshold to be reached before it acts. And, also like SIOC, the minimum is 5 ms.

While it is too high in general to be useful for FlashArray induced latency, SDRS differs from SIOC in the latency it actually looks at. SDRS uses the “VMObservedLatency” (referred to a GAVG in esxtop) averages from the hosts accessing the datastore. Therefore, this latency includes time spent queuing in the ESXi kernel. So, theoretically, a high-IOPS workload, with a low configured device queue depth limit, an I/O could conceivably spend 5 ms or more queuing in the kernel. In this situation Storage DRS will suggest moving a virtual machine to a datastore which does not have an overwhelmed queue.

That being said, this is still an unlikely scenario because:

1. The FlashArray empties out the queue fast enough that a workload must be quite intense to fill up an ESXi queue so much that is spends 5 ms or more in it. Usually, with a workload like that, the queuing is higher up the stack. (in the virtual machine)
2. Storage DRS samples for 16 hours before it makes a recommendation, so typically you will get one recommendation set per-day for a datastore. So this workload must be consistently and extremely high, for a long time, before SDRS acts.

In short, SDRS is fully supported by Pure Storage, but Pure Storage makes no specific recommendations for performance based move configuration.

Datastore Capacity Management

Managing the capacity usage of your VMFS datastores is an important part of regular care in your virtual infrastructure. There are a variety of mechanisms inside of ESXi and vCenter to monitor capacity. Frequently, the concept of data reduction on the FlashArray is seen as a complicating factor, when in reality it is a simplifying factor, or at worse, a non-issue.

Let’s overview some concepts on how to best manage VMFS datastores from a capacity perspective.

VMFS Usage vs. FlashArray Volume Capacity

VMFS reports how much is currently allocated in the filesystem on that volume. Depending on the type of virtual disk (thin or thick), dictates how much is consumed upon creation of the virtual machine (or virtual disk specifically). Thin disks only allocate what the guest has actually written to, and therefore VMFS only records what the virtual machine has written in its space usage. Thick type virtual disks allocate the full virtual disk immediately, so VMFS records much more space as being used than is actually used by the virtual machines.

This is one of the reasons thin virtual disks are preferred—you get better insight into how much space the guests are actually using.

Regardless of what type you choose, ESXi is going to take the sum total of the allocated space of your virtual disks and compare that to the total capacity of the filesystem of the volume. The used space is the sum of those virtual disks allocations. This number increases as virtual disks grow or new ones are added, and can decrease as old ones are deleted, moved, or even shrunk.

Compare this to what the FlashArray reports for capacity. What the FlashArray reports for volume usage is NOT the amount used for that volume. What the FlashArray reports is the unique footprint of the volume on that array.

In the example below we can see that we are using a 5 TB FlashArray volume and VMFS datastorea. The example confirms that the VMFS datastore reports a total of 720.72 GB of used space on the 5TB filesystem. This tell us that there is a combined total of 720.72 GB of allocated virtual disks on this filesystem:

Now let’s look at the FlashArray volume.

The FlashArray volume shows that 50.33 GB is being used. Does this mean that VMFS is incorrect? No. VMFS is always the source of truth. The “Volumes” metric on the FlashArray simply represents the amount of physical capacity that has been written to the volume after data reduction that no other volume shares.

This metric can change at any time as the data set changes on that volume or any other volume on the FlashArray. If, for instance, some other host writes 2 GB to another volume (let’s call it “volume2”), and that 2 GB happens to be identical to 2 GB of that 50.33 GB GB on “sn1-m20-e05-28-prod-ds”, then  “sn1-m20-e05-28-prod-ds” would no longer have 50.33 GB of unique space. It would drop down to 48.33 GB, even though nothing changed on “sn1-m20-e05-28-prod-ds” itself. Instead, another application just happened to write similar data, making the footprint of “sn1-m20-e05-28-prod-ds” less unique.

For a more detailed conversation around this, refer to this blog post:

http://www.codyhosterman.com/2017/01/vmfs-capacity-monitoring-in-a-data-reducing-world/

Why doesn’t VMFS report the same used capacity as the FlashArray for the underlying volume?

Well, because they mean different things. VMware reports what is allocated on the VMFS and the FlashArray reports what is unique to the underlying volume. The FlashArray value can change constantly. The FlashArray metric is only meant to show how reducible the data on that volume is internal to the volume and against the entire array. Conversely, VMFS capacity usage is based solely on how much capacity is allocated to it by virtual machines. The FlashArray volume space metric, on the other hand, actually relates to what is also being used on other volumes. In other words, VMFS usage is only affected by data on the VMFS volume itself. The FlashArray volume space metric is affected by the data on all of the volumes. So the two values should not be conflated.

For capacity tracking, you should refer to the VMFS usage. How do we best track VMFS usage? What do we do when it is full?

Monitoring and Managing VMFS Capacity Usage

As virtual machines grow and as new ones are added, the VMFS volume they sit on will slowly fill up. How to respond and to manage this is a common question.

In general, using a product like vRealize Operations Manager (vROps) with the FlashArray Management Pack is a great option here. But for the purposes of this document we will focus on what can be done inside of vCenter alone.

You need to decide on a few things:

• At what percentage full of my VMFS volume do I become concerned?
• When that happens what should I do?
• What capacity value should I monitor on the FlashArray?

The first question is the easiest to answer. Choose either a percentage full or at a certain capacity free. Do you want to do something when, for example, a VMFS volume hits 75% full or when there is less than 50 GB free? Choose what makes sense to you.

vCenter alerts are a great way to monitor VMFS capacity automatically. There is a default alert for datastore capacity, but it does not do anything other than tag the datastore object with the alarm state. Pure Storage recommends creating an additional alarm for capacity that executes some type of additional action when the alarm is triggered.

Configuring a script to run, an email to be issued, or a notification trap to be sent greatly diminishes the chance of a datastore running out of space unnoticed.

The next step is to decide what happens when a capacity warning occurs.

There are a few options:

1. Increase the capacity of the volume
2. Move virtual machines off of the volume
3. Add a new volume

Your solution may be one of these options or a mix of all three. Let’s quickly walk through the options.

Option 1: Increase the capacity of the volume

This is the simplest option. If capacity has crossed the threshold you have specified, increase the volume capacity to clear the threshold.

The process is:

1. Increase the FlashArray volume capacity.

2. Rescan the hosts that use the datastore.

3. Increase the VMFS to use the new capacity.

4. Choose “Use ‘Free space xxx GB/TB’ to expand the datastore” .

5. This will clear the alarm and add additional capacity.

Option 2: Move virtual machine off of the volume

Another option is to move one or more virtual machines from a more-full datastore to a less-full datastore. While this can be manually achieved through case-by-case Storage vMotion, Pure Storage recommends leveraging Storage DRS to automate this. Storage DRS provides, in addition to the performance-based moves discussed earlier in this document, the ability to automatically Storage vMotion VMs based on capacity usage of VMFS datastores. If a datastore reaches a certain percent full, SDRS can automatically move, or make recommendations for, virtual machines to be moved to balance out space usage across volumes.

1. SDRS is enabled on a datastore cluster.

2. When a datastore cluster is created you can enable SDRS and choose capacity threshold settings, which can either be a percentage or a capacity amount.

Pure Storage has no specific recommendations for these values and can be decided upon based on your own environment. Pure Storage does have a few recommendations for datastore cluster configuration in general:

• Only include datastores on the same FlashArray in a given datastore cluster. This will allow Storage vMotion to use the VAAI XCOPY offload to accelerate the migration process of virtual machines and greatly reduce the footprint of the migration workload.
• Include datastores with similar configurations in a datastore cluster. For example, if a datastore is replicated on the FlashArray, only include datastores that are replicated in the same FlashArray protection group so that a SDRS migration does not violate required protection for a virtual machine.

Option 3: Create a new VMFS volume

The last option is to create an entirely new VMFS volume. You might decide to do this for a few reasons:

• The current VMFS volumes have maxed out possible capacity (64 TB each).
• The current VMFS volumes have overloaded the queue depth inside of every ESXi server using it. Therefore, they can be grown in capacity, but cannot provide any more performance due to ESXi limits.

In this situation follow the standard VMFS provisioning steps for a new datastore. Once the creation of volumes and hosts/host groups and the volume connection is complete, the volumes will be accessible to the ESXi host(s). Using the vSphere Web Client, initiate a “Rescan Storage…” to make the newly-connected Pure Storage volume(s) fully-visible to the ESXi servers in the cluster. One can then use the “Add Storage” wizard to format the newly added volume.

Shrinking a Volume

For additional details on resignaturing and snapshot management, please refer to the following blog posts:

• Mounting an unresolved VMFS
• Why not force mount?
• Why might a VMFS resignature operation fail?
• How to correlate a VMFS and a FlashArray volume
• How to snapshot a VMFS on the FlashArray
• Restoring a single VM from a FlashArray snapshot

Mounting a Snapshot Volume

The Pure Storage FlashArray provides the ability to take local or remote point-in-time snapshots of volumes which can then be used for backup/restore and/or test/dev. When a snapshot is taken of a volume containing VMFS, there are a few additional steps from both the FlashArray and vSphere sides to be able to access the snapshot point-in-time data.

When a FlashArray snapshot is taken, a new volume is not created—essentially it is a metadata point-in-time reference to data blocks on the array that reflect that moment’s version of the data. This snapshot is immutable and cannot be directly mounted. Instead, the metadata of a snapshot has to be “copied” to an actual volume which then allows the point-in-time, which was preserved by the snapshot metadata, to be presented to a host. This behavior allows the snapshot to be re-used again and again without changing the data in that snapshot. If a snapshot is not needed more than one time an alternative option is to create a direct snap copy from one volume to another—merging the snapshot creation step with the association step.

When a volume hosting a VMFS datastore is copied via array-based snapshots, the copied VMFS datastore is now on a volume that has a different serial number than the original source volume. Therefore, the VMFS will be reported as having an invalid signature since the VMFS datastore signature is a hash partially based on the serial of the hosting device. Consequently, the device will not be automatically mounted upon rescan—instead the new datastore wizard needs to be run to find the device and resignature the VMFS datastore. Pure Storage recommends resignaturing copied volumes rather than mounting them with an existing signatures (referred to as force mounting).

Deleting a Datastore

Prior to the deletion of a volume, ensure that all important data has been moved off or is no longer needed. From the vSphere Web Client (or CLI) delete or unmount the VMFS volume and then detach the underlying device from the appropriate host(s).

After a volume has been detached from the ESXi host(s) it must first be disconnected (from the FlashArray perspective) from the host within the Purity GUI before it can be destroyed (deleted) on the FlashArray.

1. Unmount the VMFS datastore on every host that it is mounted to.

2. Detach the volume that hosted the datastore from every ESXi host that sees the volume.

3. Disconnect the volume from the hosts or host groups on the FlashArray.

4. Destroy the volume on FlashArray.