KB at your finger tips
This is one stop global knowledge base where you can learn about all the products, solutions and support features.
Storage and Backups-Purestorage
Troubleshooting FlashArray Storage Replication Adapter (SRA)
Locating logs - Linux appliance
The SRA's logs are located in:
/var/log/vmware/srm/SRAs/sha256{characterSpamHere}
This directory should have the logs referenced below.
The SRM's logs are located in:
/var/log/vmware/srm/
Locating logs - Windows appliance
The SRA 's logs are located in:
%PROGRAMDATA%\VMware\VMware vCenter Site Recovery Manager\Logs\SRAs\purestorage
Each invocation of the SRA produces one log file. Sort by Date Modified to see the commands executed in chronological order.
The SRM 's logs are located in :
%PROGRAMDATA%VMware\VMware vCenter Site Recovery Manager\Logs
Look for vmware-dr-##.log files. The file with the largest ## is the most recent. The SRM logs are useful for diagnosing problems when (a) The SRA responded correctly, but SRM still failed an operation, and (b) The SRA crashed on launch before being able to log anything. Before collecting SRM logs, be sure to quit SRM and wait a few seconds for all the logs to be flushed to disk.
The SRA installer's logs are located in:
%PROGRAMDATA%\PureStorage\PureSRAInstaller.log
Verifying Installation
To verify that the SRA has been installed correctly, do the following:
- Confirm that the SRA is listed in Programs and Features.
-
Find where SRM is installed by looking at
HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware vCenter Site Recovery Manager\InstallPathin the Registry. It is usuallyC:\Program Files\VMware\VMware vCenter Site Recovery Manager\bin. - In the SRM folder, navigate to the " storage/sra/purestorage" folder.
-
Confirm that the following files are present (filename case may vary):
command.pl PureSRA.exe PureSRA.pdb PureSRA.exe.config PureStorage.Rest.dll PureStorage.Rest.pdb Newtonsoft.Json
- Right click on PureSRA.exe and select Properties. Verify that it's the version you expect, and that the binary is signed by Pure Storage.
-
Confirm that .NET 3.5 is installed by checking the presence of:
%windir%\Microsoft.NET\Framework\v3.5
Confirming Environment
The SRA log starts by logging the SRA version and the info about the environment it runs in. The SRA expects to run as an admin, and as a 64-bit process. It will not work correctly otherwise. Look for something like to confirm this is the case:
[02/12/2015 11:03:30,Logging session for discoverArrays,V] Process is 64-Bit. [02/12/2015 11:03:30,Logging session for discoverArrays,V] Running as the administrator.
Confirming Array Pair
The SRA logs the input from SRM. Look for "Received input:" followed by an XML string near the top of the log file. In the XML string, there are usually two Connection nodes, with the id "localArray" and and "peerArray". They are the pair of array the operation is being applied on. These info are entered by the user when configuring the SRA from inside SRM. Make sure they are actually the arrays you are using for SRM. Example:
<Connection id="localArray">
<Addresses>
<Address id="spA">vss-purity-vm1.dev.purestorage.com</Address>
</Addresses>
...
</Connection>
<Connection id="peerArray">
<Addresses>
<Address id="spA">vss-purity-vm2.dev.purestorage.com</Address>
</Addresses>
...
</Connection>
Locating Errors
Each entry in the SRA log is associated with a verbosity level. Search for ", E ]" and ", W ]" in the log file to see the logged E rrors and W arnings, respectively. They are usually indicative of what went wrong. For example, the entries below indicate that the SRA could not connect to an array:
[02/11/2015 09:40:56,SRMCommandHandlerBase.cs:ConnectToInputAr<wbr/>rays,W] Connection failed to FlashArray at 10.66.50.90 using connection localArray [02/11/2015 09:40:56,SRMCommandHandlerBase.cs:ConnectToInputAr<wbr/>rays,E] "PureRestException: HttpStatusCode = 'BadRequest', RestErrorCode = 'InvalidVersion', Details = '', InnerException = ''"
If the entire operation failed, the SRA will output an error. Look for "Setting output:" followed by some XML string. You should see an Error node with an error ID, such as:
<Error code="1004">
<purestorage:PureExceptionMessage>...</purestorage<wbr/>:PureExceptionMessage>
<purestorage:LogFile>...</purestorage:LogFile>
</Error>
The meaning of the error codes are listed below. For example, 1004 stands for "array unreachable".
WarningSyncInProgress = 500, // Defined by VMware ErrorUnhandledException = 1001, ErrorUnknownCommand = 1002, ErrorPureException = 1003, ErrorArrayUnreachable = 1004, ErrorArrayUnauthorized = 1005, ErrorArrayIdNotAvailable = 1006, ErrorArrayIDMissing = 1007, ErrorBadArrayPair = 1008, ErrorVolumeNotInPGroup = 1009, ErrorCannotFindSyncStatus = 1010, ErrorCannotFindSnapshot = 1011, ErrorCannotFindVolume = 1012, ErrorTestFailoverStartInProgress = 1013, ErrorVolumeConnectionFailed = 1014, WarningCannotFindPgroup = 1015, ErrorCannotCreatePgroup = 1016, WarningDeviceAlreadyFailedOver = 1017, WarningPrepareFailoverInProgress = 1018, ErrorArrayInsufficientPermissions = 1019, WarningHostConnectionFailed = 1020, ErrorCannotCreateVolume = 1021, ErrorCannotDisconnectVolume = 1022, ErrorCannotRenameVolume = 1023, ErrorVolumeNotDisconnected = 1024, ErrorCannotDeleteVolume = 1025, ErrorCannotSnapshotPGroup = 1026, ErrorEmptyOrMissingDeviceID = 1027, WarningAlreadyPerformedPrepareRestoreReplication = 1028, WarningAlreadyPerformedRestoreReplication = 1029, WarningAlreadyPerformedPrepareReverseReplication = 1030
The "PureExceptionMessage" portion of the error will contain more specific information about why the operation failed. Examples are under the subheadings for specific errors below.
Additionally, the SRA logs the HTTP requests (URL only) and their response codes. Look for "Rest Library transcript:" in the log. In the HTTP transcript that follows, look for "PureStorage.Rest Error:". Note that many HTTP errors are benign and expected (e.g. we test for the existence of a volume by asking the array about it; if the array responds with a does-not-exist error, we know it doesn't exist), so view errors in the HTTP transcript in the context of other clues in the log.
Error 1004 (ErrorArrayUnreachable)
Common Problems
"invalid credentials"
Example error:
[01/04/2018 11:54:01,DiscoverArrays.cs:ProcessCommand,V] Exiting
Setting output: <?xml version="1.0" encoding="utf-8"?>
<Response xmlns="http://www.vmware.com/srm/sra/v2" xmlns:purestorage="http://www.purestorage.com/sra">
<Error code="1004">
<purestorage:PureExceptionMessage>The remote server returned an error: (400) Bad Request.
Message from Purity='ctx:,msg:invalid credentials'</purestorage:PureExceptionMessage>
<purestorage:LogFile>C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\Logs\SRAs\purestorage\discoverArrays_2018-01-04-11-53-58-443789-6607c4a5-ed27-4f4a-8d3b-46c8db303e85.log</purestorage:LogFile>
</Error>
</Response>
This usually means the array managers are configured incorrectly. If it is a "one to many" or "many to one" configuration, ensure that the Pure FlashArray username and password are the same for each array.
Error 1004 (ErrorArrayUnreachable)
Common Problems
"invalid credentials"
Example error:
[01/04/2018 11:54:01,DiscoverArrays.cs:ProcessCommand,V] Exiting
Setting output: <?xml version="1.0" encoding="utf-8"?>
<Response xmlns="http://www.vmware.com/srm/sra/v2" xmlns:purestorage="http://www.purestorage.com/sra">
<Error code="1004">
<purestorage:PureExceptionMessage>The remote server returned an error: (400) Bad Request.
Message from Purity='ctx:,msg:invalid credentials'</purestorage:PureExceptionMessage>
<purestorage:LogFile>C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\Logs\SRAs\purestorage\discoverArrays_2018-01-04-11-53-58-443789-6607c4a5-ed27-4f4a-8d3b-46c8db303e85.log</purestorage:LogFile>
</Error>
</Response>
This usually means the array managers are configured incorrectly. If it is a "one to many" or "many to one" configuration, ensure that the Pure FlashArray username and password are the same for each array.
Common Problems
"invalid credentials"
Example error:
[01/04/2018 11:54:01,DiscoverArrays.cs:ProcessCommand,V] Exiting
Setting output: <?xml version="1.0" encoding="utf-8"?>
<Response xmlns="http://www.vmware.com/srm/sra/v2" xmlns:purestorage="http://www.purestorage.com/sra">
<Error code="1004">
<purestorage:PureExceptionMessage>The remote server returned an error: (400) Bad Request.
Message from Purity='ctx:,msg:invalid credentials'</purestorage:PureExceptionMessage>
<purestorage:LogFile>C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\Logs\SRAs\purestorage\discoverArrays_2018-01-04-11-53-58-443789-6607c4a5-ed27-4f4a-8d3b-46c8db303e85.log</purestorage:LogFile>
</Error>
</Response>
This usually means the array managers are configured incorrectly. If it is a "one to many" or "many to one" configuration, ensure that the Pure FlashArray username and password are the same for each array.
"invalid credentials"
Example error:
[01/04/2018 11:54:01,DiscoverArrays.cs:ProcessCommand,V] Exiting
Setting output: <?xml version="1.0" encoding="utf-8"?>
<Response xmlns="http://www.vmware.com/srm/sra/v2" xmlns:purestorage="http://www.purestorage.com/sra">
<Error code="1004">
<purestorage:PureExceptionMessage>The remote server returned an error: (400) Bad Request.
Message from Purity='ctx:,msg:invalid credentials'</purestorage:PureExceptionMessage>
<purestorage:LogFile>C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\Logs\SRAs\purestorage\discoverArrays_2018-01-04-11-53-58-443789-6607c4a5-ed27-4f4a-8d3b-46c8db303e85.log</purestorage:LogFile>
</Error>
</Response>
This usually means the array managers are configured incorrectly. If it is a "one to many" or "many to one" configuration, ensure that the Pure FlashArray username and password are the same for each array.
Tunable: HTTP Timeout
If the SRA does not hear a response from the array with 60 seconds for a REST call, it times out (indicated by verbiage about "timed out" in the logs).
To change the HTTP timeout value from the default value of 60 seconds, do the following (on all machines where the SRA is installed):
- Launch the Registry Editor.
-
Create the registry key
HKEY_LOCAL_MACHINE\SOFTWARE\PureStorage\SRA(or navigate to it if it already exists). - Create a DWORD Value named "HTTPTimeoutInSeconds", if it doesn't exist already.
- Change the value to the desired HTTP timeout value (e.g. 120 seconds, in decimal), and press OK.
The change will take effect the next time the SRA is invoked.
Tunable: Host Connection on SRM Failover
By default, the SRA prioritizes host group connections when asked by SRM to connect to hosts (e.g. if a hostgroup HG contains a host H, the SRA will connect to HG when asked to connect to H). Most users should use this behavior.
However, if the user wishes to disable this behavior (
i.e. only connect to hosts on failover)
, the can add a
DWORD
Value (named "DisableHostGroupConnectionOnFailover") under the registry key
HKEY_LOCAL_MACHINE\SOFTWARE\PureStorage\SRA
, and set its value to 1.
Miscellaneous
One useful tool of debugging problems is to use Fiddler. You will need to follow Decrypting HTTPS-protected traffic to set up debugging for HTTPS traffic. Repeat the failed SRM operation with Fiddler running to see the HTTP traffic. You can save Fiddler trace to file and give to the dev team for further debugging.
Another noteworthy reminder is that the user needs to rescan for the SRA after upgrading it (it's covered in the manual accompanying the SRA), or they may see errors.
Additionally, it might be useful to enable a higher level of logging if what you have is not providing sufficient information. Please note that greater detail in logging can fill up whatever the logs are writing to, so you need to be really careful when enabling this and should only enable it for short periods of time while the customer is reproducing an issue and then disable it immediately after. More details can be found in this VMware KB.
Troubleshooting vSphere Plugin SSL Certificates for Browsers
Problem
When you are using the vCenter Pure plug-in you are able to view and manage the GUI within the vSphere Web Client. Each browser has a different way to handle the SSL Certificate authentication. We have a known issue with mismatched addresses that we plan to fix in the long term, in the meantime you can find the workarounds below.
Note: Certificates used with a FlashArray must be PEM formatted ( Base64 encoded).
Solution
Internet Explorer
When logging into the vCenter Web GUI on IE you may see the following:
In order to use the vSphere Plug-In you will need to perform the following:
Step 1: Disable the Option for "Warn about certificate address mismatch*" (IE11+ Only)
Step 2: Add the array's URL to the trusted sites
Step 3: Navigate to the PureStorage FlashArray's GUI URL
Do not log in, click on "Certificate Error > View Certificates"
Step 4: Install the Certificate
Once the certificate is installed, you should be able to see the Pure GUI in the vCenter Plug-in.
Google Chrome
When you log into the vSphere GUI you will see the following:
To fix this you will need to log into the array directly first & by pass the certificate warning:
Once you do this, you will be able to reload the vSphere GUI and see the Pure GUI in the plug-in.
Firefox
When logging into the vCenter Web GUI you will get the following error:
To fix this, log directly into the Pure GUI and add it as an exception:
Once you've done this you're all set as this will add it as a permanent exception.
Internet Explorer
When logging into the vCenter Web GUI on IE you may see the following:
In order to use the vSphere Plug-In you will need to perform the following:
Step 1: Disable the Option for "Warn about certificate address mismatch*" (IE11+ Only)
Step 2: Add the array's URL to the trusted sites
Step 3: Navigate to the PureStorage FlashArray's GUI URL
Do not log in, click on "Certificate Error > View Certificates"
Step 4: Install the Certificate
Once the certificate is installed, you should be able to see the Pure GUI in the vCenter Plug-in.
Step 1: Disable the Option for "Warn about certificate address mismatch*" (IE11+ Only)
Step 2: Add the array's URL to the trusted sites
Step 3: Navigate to the PureStorage FlashArray's GUI URL
Do not log in, click on "Certificate Error > View Certificates"
Step 4: Install the Certificate
Once the certificate is installed, you should be able to see the Pure GUI in the vCenter Plug-in.
Google Chrome
When you log into the vSphere GUI you will see the following:
To fix this you will need to log into the array directly first & by pass the certificate warning:
Once you do this, you will be able to reload the vSphere GUI and see the Pure GUI in the plug-in.
Firefox
When logging into the vCenter Web GUI you will get the following error:
To fix this, log directly into the Pure GUI and add it as an exception:
Once you've done this you're all set as this will add it as a permanent exception.
Verifing that ATS is Configured on a Datastore in a VMware Support Bundle
Confirming the SCSI-2 Reservations are Happening
If VMware is not configured as per best practice expectations (ATS Enabled) then we may see SCSI-2 Reservations in our logs. This is how you can check to see if that's happening:
-
Run this command
tgrep -c 'vol.pr_cache inserting registration' core*
on Penguin Fuse on the date directory in question for the array, this will give you the number of SCSI-2 reservations created every hour:
quelyn@i-9000a448:/logs/del-valle.k12.tx.us/dvisd-pure01-ct1/2015_10_27$ tgrep -c 'vol.pr_cache inserting registration' core* core.log-2015102700.gz:1875 core.log-2015102701.gz:1798 core.log-2015102702.gz:1827 core.log-2015102703.gz:1817 core.log-2015102704.gz:1860 core.log-2015102705.gz:1812 core.log-2015102706.gz:1818 core.log-2015102707.gz:2577 core.log-2015102708.gz:8181 core.log-2015102709.gz:15131 core.log-2015102710.gz:21826 core.log-2015102711.gz:19140 core.log-2015102712.gz:12044 core.log-2015102713.gz:13451 core.log-2015102714.gz:22995 core.log-2015102715.gz:33136 core.log-2015102716.gz:18587 core.log-2015102717.gz:5900 core.log-2015102718.gz:7324 core.log-2015102719.gz:2541 core.log-2015102720.gz:2213 core.log-2015102721.gz:1850 core.log-2015102722.gz:1807 core.log-2015102723.gz:1851
-
Running this command without the '-c' will allow you to see which LUNs are being locked, seen below bolded. This can yield many lines of output, so you may want to do this per log file:
quelyn@i-9000a448:/logs/del-valle.k12.tx.us/dvisd-pure01-ct1/2015_10_27$ tgrep 'vol.pr_cache inserting registration' core* core.log-2015102700.gz:Oct 26 23:18:51.394 7FB1ACCF3700 I vol.pr_cache inserting registration, seq 3097137672 vol 69674 i_t 20000025B5AA000E-spc2-0 res_type 15 core.log-2015102700.gz:Oct 26 23:18:51.431 7FB1AD5F5700 I vol.pr_cache inserting registration, seq 3097137673 vol 69673 i_t 20000025B5BB000E-spc2-0 res_type 15 core.log-2015102700.gz:Oct 26 23:18:51.457 7FB1AE378700 I vol.pr_cache inserting registration, seq 3097137674 vol 69662 i_t 20000025B5AA000E-spc2-0 res_type 15 core.log-2015102700.gz:Oct 26 23:18:51.483 7FB1AE378700 I vol.pr_cache inserting registration, seq 3097137675 vol 69661 i_t 20000025B5AA000E-spc2-0 res_type 15 core.log-2015102700.gz:Oct 26 23:18:51.503 7FB1A73FC700 I vol.pr_cache inserting registration, seq 3097137676 vol 69676 i_t 20000025B5AA000E-spc2-0 res_type 15 core.log-2015102700.gz:Oct 26 23:18:51.522 7FB1AF7FD700 I vol.pr_cache inserting registration, seq 3097137677 vol 69667 i_t 20000025B5BB000E-spc2-0 res_type 15
-
You can then run the
pslun
command to determine the volume name:
quelyn@i-9000a448:/logs/del-valle.k12.tx.us/dvisd-pure01-ct1/2015_10_27$ pslun Volume Name pslun Name -------------------------------- ---------- PURE-STR-LUN01 pslun69648 PURE-STR-LUN02 pslun69660 PURE-STR-LUN03 pslun69661 PURE-STR-LUN04 pslun69662 PURE-STR-LUN05 pslun69664 PURE-STR-LUN06 pslun69665 PURE-STR-LUN07 pslun69666 PURE-STR-LUN08 pslun69667 PURE-STR-LUN09 pslun69668 PUR-STR-LUN10 pslun69669 PUR-STR-LUN11 pslun69670 PURE-STR-LUN12 pslun69671 PURE-STR-LUN13 pslun69672 PURE-STR-LUN14 pslun69673 PURE-STR-LUN15 pslun69674 PURE-STR-LUN16 pslun69675 PURE-STR-LUN17 pslun69676 PURE-STR-LUN18 pslun69677 PURE-STR-LUN19 pslun69678
- To determine which hosts are creating the SCSI-2 Reservations, we will need a VMware bundle. The customer can send this to us via FTP.
- Once the bundle is uploaded, please prepare for analysis as per KB: Retrieving Customer Logs from the FTP
-
Run the vm script that jhop created against the bundles to check the global configuration of VAAI ATS Wiki: VMware vSphere Ovreview and Troubleshooting
/home/jhop/python/Mr_VMware.py
- After we have confirmed that VAAI ATS is enabled globally if we are still seeing SCSI-2 reservations we will want to check each volume individually. Please proceed to the next section:
Identifying datastore ATS Configuration on VMware ESXi
Step 1:
Since we only care about datastores (not Raw Device Maps (RDMs)) on Pure Storage we will find our applicable LUNs in the "esxcfg-scsidevs -m.txt" file under the "commands" folder in a VMware Support Bundle. Below is an example of what a line from there will look like:
There are several things that we want to identify from this output; the first is the "NAA Identifier". This is important because anything starting with "naa.624a937" is a Pure Storage LUN. Once we have a Pure Storage LUN we then want to take note of the "VMFS UUID" number (i.e. 53c80075-7ddcc5ba-7d03-0025b5000080 ). The reason why we focus on this instead of the "User-Friendly Name" is because the customers can choose any name they want in that option. If we choose the VMFS UUID then we are guaranteed to know we are referring to a Pure Storage LUN since that is a uniquely generated ID that the vCenter Server assigns to individual LUNs.
Step 2:
Once we have this information the next step is to search for the "vmkfstools" text file that contains the File System information on this device; this will also be found in the "commands" folder you already reside in. An example of what the text file will look like is as follows:
vmkfstools_-P--v-10-vmfsvolumes 53c80075-7ddcc5ba-7d03-0025b5000080 .txt
Notice above our "VMFS UUID" is contained in the file name (in red). We can now search this file for the "Mode" it is running in. An example of what this line will look like, if configured properly, is as follows:
Mode: public ATS-only
If the datastore is not configured properly it will look as follows:
Mode: public
If the datastore is showing a "public" mode then we know that this datastore is misconfigured and we'll be receiving an excessive amount of SCSI-2 Reservations from the ESXi Hosts. This means that locking tasks are not being offloaded to the FlashArray.
Obviously if the customer has a lot of LUNs this process above can take a while, so it is best to script this. I have listed below a simple one liner that will do this for you if you would like to use this instead of going through each LUN one-by-one:
grep "naa.624a937" esxcfg-scsidevs_-m.txt | awk '{print $3}' > Pure-LUNs.txt;while read f;do cat vmkfs*$f.txt |grep -e "Mode:" -e "naa.624a937";echo;done < Pure-LUNs.txt
NOTE: This command is able to be copied & pasted then used on any ESXi Host that is 5.0 and higher, as long as you are in the "commands" folder of the ESXi Host you want to verify.
Step 1:
Since we only care about datastores (not Raw Device Maps (RDMs)) on Pure Storage we will find our applicable LUNs in the "esxcfg-scsidevs -m.txt" file under the "commands" folder in a VMware Support Bundle. Below is an example of what a line from there will look like:
There are several things that we want to identify from this output; the first is the "NAA Identifier". This is important because anything starting with "naa.624a937" is a Pure Storage LUN. Once we have a Pure Storage LUN we then want to take note of the "VMFS UUID" number (i.e. 53c80075-7ddcc5ba-7d03-0025b5000080 ). The reason why we focus on this instead of the "User-Friendly Name" is because the customers can choose any name they want in that option. If we choose the VMFS UUID then we are guaranteed to know we are referring to a Pure Storage LUN since that is a uniquely generated ID that the vCenter Server assigns to individual LUNs.
Step 2:
Once we have this information the next step is to search for the "vmkfstools" text file that contains the File System information on this device; this will also be found in the "commands" folder you already reside in. An example of what the text file will look like is as follows:
vmkfstools_-P--v-10-vmfsvolumes 53c80075-7ddcc5ba-7d03-0025b5000080 .txt
Notice above our "VMFS UUID" is contained in the file name (in red). We can now search this file for the "Mode" it is running in. An example of what this line will look like, if configured properly, is as follows:
Mode: public ATS-only
If the datastore is not configured properly it will look as follows:
Mode: public
If the datastore is showing a "public" mode then we know that this datastore is misconfigured and we'll be receiving an excessive amount of SCSI-2 Reservations from the ESXi Hosts. This means that locking tasks are not being offloaded to the FlashArray.
Obviously if the customer has a lot of LUNs this process above can take a while, so it is best to script this. I have listed below a simple one liner that will do this for you if you would like to use this instead of going through each LUN one-by-one:
grep "naa.624a937" esxcfg-scsidevs_-m.txt | awk '{print $3}' > Pure-LUNs.txt;while read f;do cat vmkfs*$f.txt |grep -e "Mode:" -e "naa.624a937";echo;done < Pure-LUNs.txt
NOTE: This command is able to be copied & pasted then used on any ESXi Host that is 5.0 and higher, as long as you are in the "commands" folder of the ESXi Host you want to verify.
Resolution
Once we have the misconfigured LUNs identified the customer can use the VMware KB listed below to resolve the issue:
Link to VMware KB: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033665
Follow the steps outlined in the "Changing an ATS-only volume to public". Simply change the "0" they are setting to a "1" in the listed command they provide to turn ATS-only back on. It is important that the customer read all of the steps before continuing forward and reading the notes & caveats.
Alternatively, and also much less of a headache, the customer can simply create a new LUN from the FlashArray and mount it to the applicable ESXi Host(s). Once the new VMFS datastore is created they can verify that ATS is properly configured. Once confirmed the new datastore has ATS enabled can then migrate the Virtual Machines from the misconfigured datastore to the newly configured datastore. After everything has been moved from the old datastore and they have confirmed all is working well, they can simply destroy the old LUN. This is much easier to do and is typically what should be recommended as the first step.
If there are any questions please reach out to a fellow colleague or Support Escalations team member for assistance.
Troubleshooting: Could Not Generate DH Keypair
Symptoms
When attempting to install the Pure Storage vSphere plugin in a vSphere 5.1 environment, the following errors are reported in the vSphere virgo client logs:
[2016-02-11 16:06:03.213] ERROR [ERROR] http-bio-9443-exec-16 com.purestorage.FlashArrayHelper javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair
and
Caused by: java.security.InvalidAlgorithmParameterException: Prime size must be multiple of 64, and can only range from 512 to 1024 (inclusive
We have also seen this behavior in customer environments were the plugin was previously working but stopped functioning. In an environment where the plugin has been working previously, but is now failing with the same errors above, are all caused by the same issue.
Diagnosis
Due to JAVA security changes for "mod_ssl" , Diffie-Hellman (DH) parameters now include primes with lengths of more than 1024 bits. Since Java 7 and earlier limit their support for DH prime sizes to a maximum of 1024 bits, the SSL negotiation between our FlashArray and vCenter fail. This issue is not caused by the FlashArray (or our vSphere Plugin) but is a problem with vCenter 5.1 using Java Development Kit (JDK) 6.
This issue directly correlates with Oracle Bug ID JDK-7044060. You can also get additional information from this issue here.
Solution
-
Go to the Oracle Archive site to
download & install
Java Development Kit 7 Update 17 on the vSphere web client server.
-
Find and make a copy of the file "wrapper.conf" from the following location: C:\Program Files\VMware\Infrastructure\vSphereWebClient\server\bin\service\conf\wrapper.conf (right click on it, select copy to desktop)
-
Edit the vSphere Web Client "wrapper.conf" with the following changes:
-
Modify
"wrapper.java.command" with the path to the new JDK 1.7U17 path.
-
32-bit version:
wrapper.java.command=C:/Program Files (x86)/Java/jdk1.7.0_17/bin/java -
64-bit version:
wrapper.java.command=C:/Program Files/Java/jdk1.7.0_17/bin/java
-
32-bit version:
-
Add
the following lines at the top of the wrapper.conf file:
-
32-bit version:
set.default.JAVA_HOME=C:\Program Files (x86)\Java\jdk1.7.0_17 -
64-bit version:
set.default.JAVA_HOME=C:\Program Files\Java\jdk1.7.0_17set.default._JAVA_OPTIONS=-Xmx1024M
-
32-bit version:
-
Comment out
(with a hash (#)) the following two lines in the wrapper.conf file:
wrapper.java.initmemory=1024m
wrapper.java.maxmemory=2048m
-
Modify
"wrapper.java.command" with the path to the new JDK 1.7U17 path.
-
R estart the vSphere Web Client service.
The path to the wrapper.conf file can be found in the following location on the vCenter Server: C:\Program Files\VMware\Infrastructure\vSphereWebClient\server\bin\service\conf\wrapper.conf
Please only make the changes to the file based on whether or not they downloaded the 32-bit or 64-bit version of the JDK
For clarify purposes, below is the top half of a wrapper.conf file showing where all the applicable changes have been made:
#******************************************************************** # Wrapper License Properties (Ignored by Community Edition) #******************************************************************** # Include file problems can be debugged by removing the first '#' # from the following line: ##include.debug #encoding=UTF-8 wrapper.license.type=DEV wrapper.license.id=201106200012 wrapper.license.licensee=VMware Global, Inc. wrapper.license.dev_application=vSphere Web Client wrapper.license.features=pro, 64bit wrapper.license.upgrade_term.begin_date=2009-10-27 wrapper.license.upgrade_term.end_date=2012-01-27 wrapper.license.key.1=feca-7df5-2263-9092 wrapper.license.key.2=a38a-acfa-38de-8031 wrapper.license.key.3=c824-a8fa-b95a-1b89 wrapper.license.key.4=8434-7a46-4450-d081 ####################################################################################################### ## You must set the SERVER_HOME property either in your environment of here before running as a service ####################################################################################################### #set.default.JAVA_HOME=<set JAVA_HOME> set.default.SERVER_HOME=<set SERVER_HOME> set.default.CONFIG_DIR=%SERVER_HOME%/config set.default.JMX_PORT=9875 set.default.JAVA_HOME=C:\Program Files (x86)\Java\jdk1.7.0_17 <------------- ADDED Line set.default._JAVA_OPTIONS=-Xmx1024M <------------- ADDED Line ######### # General ######### wrapper.console.title=vSphere Web Client #wrapper.debug=TRUE ############# # Application ############# wrapper.java.command=C:/Program Files (x86)/Java/jdk1.7.0_17/bin/java <------ Modified Line wrapper.working.dir=%SERVER_HOME% wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp ########### # Classpath ########### wrapper.java.classpath.1=%SERVER_HOME%/bin/service/lib/wrapper.jar wrapper.java.classpath.2=%SERVER_HOME%/lib/*.jar ############## # Library Path ############## wrapper.java.library.path.1=%SERVER_HOME%/bin/service/lib ############# # JVM Memory ############# #wrapper.java.initmemory=1024m <------------- Commented out line #wrapper.java.maxmemory=1024m <------------- Commented out line