Relevant ESXi Versions: 5.5 Update 2 or ESXi 6.0
ESXi Hosts which support VAAI ATS Heartbeat, may cause the host to lose connectivity to datastores under certain conditions.
Problem symptoms
An ESXi 5.5 Update 2 or ESXi 6.0 host loses connectivity to a VMFS5 datastore, while using the VAAI ATS heartbeat.
In the /var/run/log/vobd.log file, you see the VOB message:
Lost access to volume <uuid><volume name> due to connectivity issues. Recovery attempt is in progress and the outcome will be reported shortly
In the /var/run/log/vmkernel.log file, you see the message:
ATS Miscompare detected between test and set HB images at offset XXX on vol YYY
In the /var/log/vmkernel.log file, you see similar error messages indicating an ATS miscompare:
2015-11-20T22:12:47.194Z cpu13:33467)ScsiDeviceIO: 2645: Cmd(0x439dd0d7c400) 0x89, CmdSN 0x2f3dd6 from world 3937473 to dev "naa.50002ac0049412fa" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.
You may also see:
- Hosts disconnecting from vSphere vCenter
- Virtual machines hanging on I/O operations
Notes:
- For VMFS5 datastores, ATS heartbeat setting is on by default.
- For VMFS3 datastores, ATS heartbeat setting is off by default.
Resolution
(!) Note - These operations can be safely performed online, while the storage is in use.
For VMFS5 and VMFS6 Datastores
To disable ATS heartbeat, run the following command from the CLI command:
# esxcli system settings advanced set -i 0 -o /VMFS3/UseATSForHBOnVMFS5
To review the results of changing options, run this command:
# esxcli system settings advanced list -o /VMFS3/UseATSForHBonVMFS5
The expected output will be similar to:
Path: /VMFS3/UseATSForHBOnVMFS5
Type: integer
Int Value: 0
Default Int Value: 1
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: Use ATS for HB on ATS supported VMFS5 volumes
References:
- VMware KB - ESXi host loses connectivity to a VMFS3 and VMFS5 datastore (2113956)