Welcome to Knowledge Base!

Problem

This guide shows how to re-scan and examine FC LUNS on Solaris 10/11. This is required for detecting new LUNs and to assist with basic storage connectivity troubleshooting.

Impact

Re-scanning FC LUNs on Solaris is generally non-disruptive in a well configured environment with normal load.

Solution

root@Unixarena-SOL11:~# luxadm -e port |grep CONNECTED
/devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl CONNECTED
/devices/pci@1d,700000/SUNW,qlc@3/fp@0,0:devctl CONNECTED

Unix@sol# cfgadm -al -o show_FCP_dev |grep fc-fabric
c2                             fc-fabric    connected    configured   unknown
c4                             fc-fabric    connected    configured   unknown
Unix@sol#

Unix@sol# fcinfo hba-port |grep Port
HBA Port WWN: 10000000c884bb48
HBA Port WWN: 10000000c884bb49
HBA Port WWN: 10000000c884b85c
HBA Port WWN: 10000000c884b85d
Unix@sol#

Unix@sol# luxadm -e dump_map /dev/cfg/c4
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    29900   0        50080e8008cfb814 50080e8008cfb814 0x0  (Disk device)
1    27400   0        10000000c884b85c 20000000c884b85c 0x1f (Unknown Type,Host Bus Adapter)
Unix@sol#

Unix@sol# cfgadm -al -o show_FCP_dev c2 c4

Unix@sol# cfgadm -c configure c2::50080e8008cfb814
Unix@sol# cfgadm -c configure c4::50080e8008cfb814

root@Unixarena-SOL11:~# luxadm -e port |grep CONNECTED
/devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl CONNECTED
/devices/pci@1d,700000/SUNW,qlc@3/fp@0,0:devctl CONNECTED

root@Unixarena-SOL11:~# luxadm -e forcelip /devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl

Unix@sol# cfgadm -al -o show_FCP_dev |grep fc-fabric
c2                             fc-fabric    connected    configured   unknown
c4                             fc-fabric    connected    configured   unknown
Unix@sol#
Unix@sol# luxadm -e forcelip /dev/cfg/c2

root@Unixarena-SOL11:~# luxadm display /dev/rdsk/c1txxxxxxd0s2

Understanding How FC is Configured on a Solaris Host

The following commands are useful for general SAN stack fact-finding on Solaris:

1. List the connected HBA’s:

root@Unixarena-SOL11:~# luxadm -e port |grep CONNECTED
/devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl CONNECTED
/devices/pci@1d,700000/SUNW,qlc@3/fp@0,0:devctl CONNECTED

2. Verify FC ports are connected and configured:

Unix@sol# cfgadm -al -o show_FCP_dev |grep fc-fabric
c2                             fc-fabric    connected    configured   unknown
c4                             fc-fabric    connected    configured   unknown
Unix@sol#

3. To find the HBA’s World Wide Node number (WWN), use the fcinfo command:

Unix@sol# fcinfo hba-port |grep Port
HBA Port WWN: 10000000c884bb48
HBA Port WWN: 10000000c884bb49
HBA Port WWN: 10000000c884b85c
HBA Port WWN: 10000000c884b85d
Unix@sol#

4. You can also find the WWN using luxadm command, if HBA is already connected to the FC switch.

Unix@sol# luxadm -e dump_map /dev/cfg/c4
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    29900   0        50080e8008cfb814 50080e8008cfb814 0x0  (Disk device)
1    27400   0        10000000c884b85c 20000000c884b85c 0x1f (Unknown Type,Host Bus Adapter)
Unix@sol#

From the above output, the last line shows the HBA information. In the same way you can find the other controller information as well.

5. Zoning can be verified using the below command:

Unix@sol# cfgadm -al -o show_FCP_dev c2 c4

6. If you see any controller port WWN shown as “unconfigured," then  you can initiate FC session using below mentioned command:

Unix@sol# cfgadm -c configure c2::50080e8008cfb814
Unix@sol# cfgadm -c configure c4::50080e8008cfb814

Scanning for New FC Devices and Getting LUN Information:

Note : The command luxadm probe can also be used to scan FC LUNs.

Resetting the HBA

If you are still unable to see the new LUN/DISK, then, if you have multipathing enabled, you can try to reset the HBA.  (Do not try this in critical servers unless you are confident that multipathing is correctly configured)

1. List the connected HBA.

root@Unixarena-SOL11:~# luxadm -e port |grep CONNECTED
/devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl CONNECTED
/devices/pci@1d,700000/SUNW,qlc@3/fp@0,0:devctl CONNECTED

2. Reset the HBA using forcelip option:

root@Unixarena-SOL11:~# luxadm -e forcelip /devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl

The forcelip command can be issued to the controller names as well:

Unix@sol# cfgadm -al -o show_FCP_dev |grep fc-fabric
c2                             fc-fabric    connected    configured   unknown
c4                             fc-fabric    connected    configured   unknown
Unix@sol#
Unix@sol# luxadm -e forcelip /dev/cfg/c2

3. Verify the controller status using cfgadm -al . Make sure disks didn't loose any SAN paths after the HBA reset. If everything seems to be working, then issue the forcelip command to the other controller as well.

If you are still not able to see the new FC/SAN LUNS, then, as a last resort, reboot the server and try again. Once you see the new LUN, make sure that you also see all FC paths. A minimum two FC paths is required for SAN disks.

4. To verify the FC LUN details and multipathing, enter the following command:

root@Unixarena-SOL11:~# luxadm display /dev/rdsk/c1txxxxxxd0s2

Requirements to Install vSphere Plugin 2.5.1+ from the PureStorage GUI

For instructions on installing and updating the plugin from the Pure GUI see VMware vSphere Plugin Install (Version 3.0 | Version 2.5 or Older).

• Java 1.8 (for TLS 1.1 or 1.2). If Java 1.8+ is not present DO NOT REQUEST TO UPDATE IT.
• Network access between Array and vSphere server through TCP ports 443.
• vCenter administrator privileges and PureStorage system administrator privileges, or an administrator with these privileges is available to do the installation. Administrative username and password for vCenter server.
• vSphere Web Client.

Requirements to Install and Configure vSphere Plugin Using the PluginServer Method

• Linux OS or Windows OS (either can be VM’s).
• Network access to vSphere server port 8080 or 8081.
• Administrative username and password for vCenter server.
• Windows OS does not need java 1.8 (the pluginserver has a java run-time included).
• Linux OS needs java jdk 1.8+ and jre 8+(for TLS 1.1 or 1.2).
• The pluginserver files in archive.
• vSphere Web Client
• PureStorage system administrator privileges to connect to PureStorage array within vSphere.
• TCP Ports 443 (for vCenter to send API commands to the array).
• vSphere 5.5, vsphere 6.0 and vSphere 6.5. Note: The space reclamation feature requires vSphere 6.0 or higher and ESXi 6.0 or higher.

Troubleshooting

If after following the documentation for VMware vSphere Plugin Installation and the plugin still does not function properly, this is often due to the VMware vSphere web client server not supporting TLS 1.1 or 1.2. See Analyzing the vsphere_client_virgo.log below if you would like to confirm this.

Installing the PluginServer

The procedure below does a few different things. It runs the unregisterplugin sh/bat using the IP address of the vSphere server. It commands this server to stop its attempt to download the vSphere plugin from our array through the API and then attempts installation of the plugin. The next time a user logs into the vSphere web client, the web client triggers the plugin's installation when a user logs in.  If there was a failed attempt to prevent login from taking a long time, it will not try and install the plugin twice. Restarting the vSphere Web Client service resolves this. Next, the startserver enables the API to pull the needed files from the Windows or Linux OS. And finally, the registerplugin uses API to pull the needed files to the vSphere server and tell vSphere to install the plugin the next time someone logs into the vSphere web client.

[2016-02-11 16:06:03.213] ERROR [ERROR] http-bio-9443-exec-16         com.purestorage.FlashArrayHelper      javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair

[2017-06-19 10:34:00.954] [ERROR] vc-service-pool-2169   70002699 100142 200004 com.vmware.vise.vim.extension.VcExtensionManager    Error unzipping https://192.168.41.131/download/pure...?version=2.5.1 to directory C:\ProgramData\VMware\vSphere Web Client\vc-packages\vsphere-client-serenity\com.purestorage.plugin.vsphere-2.5.1, check if the server process has Write Permission on this machine. java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(Unknown Source)
        at java.net.SocketInputStream.read(Unknown Source)
        at sun.security.ssl.InputRecord.readFully(Unknown Source)
        at sun.security.ssl.InputRecord.read(Unknown Source)

Windows

If any of these steps do not work see the notes section below.

1. Download PureStorage_vSphere_installer.jar and the PluginServer-2.5.1_201704051804+163c8cd-rel_2_5_x.zip or latest plugin server from https://archive.dev.purestorage.com/flasharray/purity/customer_shipped_releases/older_releases_at_.._.._.._customer_shipped_releases/vsphere_plugin/ and then copy these files to a google drive (or equivalent) to share.
2. Make a new directory for the PluginServer files and unzip them to the directory.
   1. Enter make a folder in the gui called pluginserver or in CLI mkdir [drive:]path/pluginserver .
   2. Extract Pluginserver-2.5.1_201704051804+163c8cd-rel_2_5_x.zip or latest to the pluginserver folder.
3. Extract the PluginServer to above folder and make sure PureStorage_vSphere_installer.jar is present in the same folder as the unzipped PluginServer files.
   1. Enter cd pluginserver/ .
   2. Enter copy ../PureStorage_vSphere_installer.jar /pluginserver .
4. Unregister the failed plugin install.
   1. Enter unregisterplugin.bat , while in the folder where the extracted files and the PureStorage_vSphere_installer.jar resides.
   2. Enter the vSphere server'ss IP address.
   3. Enter the vSphere credentials.
5. Run startserver.bat in the command console used for creating the directory. This will start up the web server that hosts the plugin. Keep this console open and running while you perform the rest of the steps.
   1. You should see something similar to the following:

      robm$ ./startserver.bat
      Found these arguments port(8080) keystore(keystore.jks)
      File location(purestorage-vsphere-plugin.zip)
      Starting server on port 8080...
      Server started successfully!

   2. Leave this running and open a new command line window for the next step.
   3. If it doesn’t work and you see “Found these arguments port(8080)” … “Address already in use” follow the steps in the note section below.
6. In a new command window, run registerplugin.bat while in the directory where unzip extracted the pluginserver and the PureStorage_vSphere_installer.jar resides.
   1. Once run, it will ask for the IP address and credentials to run API commands against the vSphere server.
7. Restart the vSphere Web Client service on the vCenter server and then wait five minutes for it to fully come up( on vCenter 6.5, you won't see the vSphere Web Client service listed in Services, so you can use a command prompt to stop and start the process:)

C:\Program Files\VMware\vCenter Server\bin>service-control --stop vspherewebclientsvc
C:\Program Files\VMware\vCenter Server\bin>service-control --start vspherewebclientsvc

Now have the user log into the vSphere web client to see the PureStorage plugin, if it is there, you can now close both command line windows.

8. If we still do not see the PureStorage Plugin make sure they have logged out then back into the vSphere Web Client.

9. If we still do not see PureStorage Plugin proceed to Analyzing the vsphere_client_virgo.log (steps below).

Once the user has logged into vSphere Web Client and can see the plugin, follow the vSphere Web Client user guide.

Linux VCSA Appliance, Installing the Plugin Server on a Linux VM

vCenter Server Appliance VCSA requires another OS to run the pluginserver scripts on. This can be another temporary VM running Linux or Windows, the following instructions are for a linux VM. Follow the above Windows steps if this other OS is running windows OS.

If any of these steps do not work see the notes section below.

1. vCenter Server Appliance VCSA requires another machine to run the pluginserver scripts on. This can be another temporary VM running Linux or Windows (above windows steps) or can be the Windows Guest OS the customer is using for their machine to temporarily run the pluginserver scripts.
2. Download PureStorage_vSphere_installer.jar from and the PluginServer-2.5.1_201704051804+163c8cd-rel_2_5_x.zip or latest plugin server from https://archive.dev.purestorage.com/flasharray/purity/customer_shipped_releases/older_releases_at_.._.._.._customer_shipped_releases/vsphere_plugin/.
3. From the command line make new directory for the pluginserver files and unzip them to the directory.
   1. Enter mkdir pluginserver .
   2. Enter unzip -d pluginserver/ Pluginserver-2.5.1_201704051804+163c8cd-rel_2_5_x.zip .
4. Extract the PluginServer to any directory and make sure to copy PureStorage_vSphere_installer.jar to the same directory as the unzipped pluginserver files.
   1. Enter cd pluginserver/ .
   2. Enter cp ../PureStorage_vSphere_installer.jar ./ .
5. Make the scripts executable.
   1. Enter chmod a+x *.sh .
6. Unregister the failed plugin install.

   1. Enter ./unregisterplugin.bat, while in the directory where unzip extracted the files and the PureStorage_vSphere_installer.jar resides.

   2. Enter the vSphere servers ip address.
   3. Enter the vSphere credentials.
7. Execute the plugin using the pluginserver.
   1. ./startserver.sh .
   2. You should see something similar to the following:

      robm$ ./startserver.sh
      Found these arguments port(8080) keystore(keystore.jks)
      File location(purestorage-vsphere-plugin.zip)
      Starting server on port 8080...
      Server started successfully!

   3. If it doesn’t work and you see “Found these arguments port(8080)” … “Address already in use” follow the steps in the note section below.
   4. If this fails for another reason, you may need to java jdk 1.8+ and jre 8+ on this linux VM. If this is a production VM, you do not want to update java, use a nonproduction VM.
   5. In testing, a ubuntu-16.04.3 VM was used. The following commands were required:.

      robm@ubuntu:~$ sudo apt-get update
      robm@ubuntu:~$ sudo apt-get install default-jre

8. Leave this running and open a new command line window for the next step.
9. In a new command window, run registerplugin.sh while in the directory where unzip extracted the PluginServer and where the PureStorage_vSphere_installer.jar resides.
   1. Once run, the program prompts for the IP address and credentials to run API commands against the vSphere server.
10. Restart the vSphere Web Client service from the command line on the vCenter server (not the VM) and then wait five minutes for it to fully come up.
    1. Enter service vsphere-client restart .
11. Now have the user log into the vSphere web client to see the PureStorage plugin, if it is there, you can now close both command line windows.
    1. If you still do not see the PureStorage Plugin, make sure you have logged out then back into the vSphere Web Client.
    2. If you still do not see PureStorage Plugin, proceed to Analyzing the vsphere_client_virgo.log (steps below).
12. Once the user has logged into vSphere Web Client and can see the plugin, follow the vSphere Web Client user guide.

Analyzing the vsphere_client_virgo.log

Locate the vsphere_client_virgo.log and have the customer copy them to a text file then email them to us. These logs often get zipped up and numbered so be sure we have the file for the time period of when we attempted to install the plugin. Below is where these files should be located. Often this file is not where we expect it to be so searching it may be necessary. The following is from this vmware kb.

1. For vSphere 5.0, all the logs for the vSphere Web Client service are located at:
   • Windows: C:\Program Files\VMware\Infrastructure\vSphere Web Client\DMServer\serviceability\.
   • Linux: /usr/lib/vmware-vsphere-client/server/serviceability/.
2. For vSphere 5.1, all the logs for the vSphere Web Client service are located at:
   • Windows: C:\ProgramData\VMware\vSphere Web Client\serviceability\.
   • Linux: /var/log/vmware/vsphere-client/serviceability/.
3. For vSphere 5.5, all the logs for the vSphere Web Client service are located at:
   • Windows: C:\ProgramData\VMware\vSphere Web Client\serviceability\.
   • Linux: /var/log/vmware/vsphere-client/.
4. For vSphere 6.0+ as per the following VMware kb.
   • C:\ProgramData\VMware\vCenterServer\logs\vsphere-client\logs.
   • /var/log/vmware/vsphere-client/logs.

Search the vsphere_client_virgo log for when the plugin was attempted. Usually, it will say “purestorage” when the attempt to install was performed. The following are errors we have seen in this file.

[2016-02-11 16:06:03.213] ERROR [ERROR] http-bio-9443-exec-16         com.purestorage.FlashArrayHelper      javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair

This alert was from not having JDK 1.8 see this JIRA.

[2017-06-19 10:34:00.954] [ERROR] vc-service-pool-2169   70002699 100142 200004 com.vmware.vise.vim.extension.VcExtensionManager    Error unzipping https://192.168.41.131/download/pure...?version=2.5.1 to directory C:\ProgramData\VMware\vSphere Web Client\vc-packages\vsphere-client-serenity\com.purestorage.plugin.vsphere-2.5.1, check if the server process has Write Permission on this machine. java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(Unknown Source)
        at java.net.SocketInputStream.read(Unknown Source)
        at sun.security.ssl.InputRecord.readFully(Unknown Source)
        at sun.security.ssl.InputRecord.read(Unknown Source)

This alert is due to vSphere trying to establish communication with the array using TLSv1.0 ES-27873.

If installing from the GUI and it does not work and you see the above messages, install the PluginServer. If the vSphere server resides on a windows OS the PluginServer can be installed on it as described above. If the Linux VCSA appliance is being used we can put the PluginServer on a windows OS server (vm or bare metal) or a Linux vm as described above. Make sure this is not a production Linux VM as we may need to update or install java to 1.8+.

Notes:

If running the startserver.bat/sh script fails, there may be an issue when port 8080 is already in use.  As seen below:

robm$Found these arguments port(8080) keystore(deystore.jks)
File location(purestorage-vsphere-plugin.zip)
java.net.BindException: Address already in use: bind”
robm$ ./startserver.sh
Found these arguments port(8080) keystore(keystore.jks)
      File location(purestorage-vsphere-plugin.zip)
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:433)
    at sun.nio.ch.Net.bind(Net.java:425)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at sun.net.httpserver.ServerImpl.<init>(ServerImpl.java:100)
    at sun.net.httpserver.HttpsServerImpl.<init>(HttpsServerImpl.java:50)
    at sun.net.httpserver.DefaultHttpServerProvider.createHttpsServer(DefaultHttpServerProvider.java:39)
    at com.sun.net.httpserver.HttpsServer.create(HttpsServer.java:90)
    at com.purestorage.PluginServer.main(Unknown Source)
Exception in thread "main" java.lang.NullPointerException: null SSLContext
    at com.sun.net.httpserver.HttpsConfigurator.<init>(HttpsConfigurator.java:82)
    at com.purestorage.PluginServer$1.<init>(Unknown Source)
    at com.purestorage.PluginServer.main(Unknown Source)
 

To fix this error modify the startserver.bat/sh so it is using 8081.

1. Open startserver.bat/sh in a text editor (notepad or vim).
2. Search for 8080 then change it to 8081 twice.
3. Change port 8080 to 8081 in the registerplugin.bat/sh.

Here is what the startserver.bat/sh looks like after modification:

#!/bin/bash
set -e
 
if [ ! -f /usr/bin/java ]; then
   echo 'Please install java'
   exit -1
fi
 
# The listening port is 8081.  You can modify to use any port.  Make sure to modify the registerPlugin script.
java -cp ./PureStorage_PluginServer.jar com.purestorage.PluginServer 8081 keystore.jks purestorage-vsphere-plugin.zip

Additionally, you will need to edit the registerplugin.bat/sh with the following command:

java -cp ./PureStorage_PluginServer.jar:./PureStorage_vSphere_installer.jar com.purestorage.RegisterPlugin 8081 3.0.0 $ip

Additional troubleshooting steps include:

1. If you get a JNI error when running the registerserver script, most likely the installer.jar isn't in the same directory as the unzipped PluginServer scripts.
2. To determine where the plugin is failing, you can verify if it is showing up in the https://vcenterIP/mob.
   1. Go to https://ipaddress_of_vSphere_server/mob.
   2. Go to Content > ExtensionManager > extensionList["com.purestorage.plugin.vsphere"] > client .
   3. From there verify the URL and ensure that this is the correct location vCenter should be looking to download and install the plugin.
3. You can also ensure that the vCenter GUI has the plugin enabled via the following:
   1. Go to https://ipaddress_of_vSphere_server/.
   2. Go to Home > Administration > Client Plug-Ins .
   3. From there make sure the Pure Storage Plugin is enabled (and not set to disabled).

Problem

When creating a VVol datastore on a Cisco UCS Server with Fibre Channel, failures can occur when an older driver is in use. The older fnic driver cannot detect protocol endpoints as it does not support sub-luns (VVols).

Impact

The FlashArray vSphere Plugin fails to mount the VVol datastore with the error:

The following hosts do not have a valid protocol endpoint connection to the selected Pure Storage Array

Or when mounting manually, the datastore is marked on the host as inaccessible.

The /var/log/vmkernel.log file on the ESXi host will show the following VVol PE warnings when the “Rescan Storage” is initiated:

2018-01-09T18:04:42.098Z cpu5:65799)WARNING: ScsiPath: 705: Sanity check failed for path vmhba0:C0:T1:L1. The path to a VVol PE comes from adapter vmhba0 which is not PE capable. Path dropped.

The problem is likely caused by outdated scsi-fnic Cisco UCS drivers.

To check for general support for sub-luns (VVols), run the following command:

esxcli storage core adapter list

Look for Second Level Lun ID in the Capabilities column.

Solution

esxcli software vib get -n scsi-fnic

esxcli software vib install -v <full_path_to driver_file>

esxcli software vib install -v /tmp/scsi-fnic_1.6.0.37-1OEM.600.0.0.2494585.vib

Check Installed Version Of scsi-fnic Cisco UCS Driver

1. Log in to ESXi host and execute

esxcli software vib get -n scsi-fnic

Update scsi-fnic Cisco UCS Driver

To install the new driver version:

1. Download the updated driver package from http://software.cisco.com

or at

https://my.vmware.com/group/vmware/details?productId=491&downloadGroup=DT-ESX60-CISCO-FNIC-16033

2. Copy scsi-fnic vib file to the host which needs updating
3. As root execute the following command on the ESXi host:

esxcli software vib install -v <full_path_to driver_file>

Example:

esxcli software vib install -v /tmp/scsi-fnic_1.6.0.37-1OEM.600.0.0.2494585.vib

The installation result should look similar to the output below:

Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: CSCO_bootbank_scsi-fnic_1.6.0.37-1OEM.600.0.0.2494585
VIBs Removed: CSCO_bootbank_scsi-fnic_1.6.0.36-1OEM.600.0.0.2494585
VIBs Skipped: