The Delphix Engine relies on stable back-end storage attached to the VMware ESX server and presented to Delphix as virtual logical units (LUNs). In rare cases, memory contention on the ESX sever may result in recurring I/O errors, and these errors can result in filesystem corruption and/or data loss on the Delphix filesystem.
The problem can affect any release of Delphix Engine software.
Delphix Engine 18.104.22.168, 22.214.171.124, and later releases include changes that significantly reduce the impact of IO failures; however, silent IO failures from VMDK storage may still result in data corruption.
The problem is only known to occur when Delphix LUNs are comprised of VMDK disks stored on VMFS3 or VMFS5.
The problem can only occur when the total size of VMDK disks on an ESX server hosting a Delphix Engine is 4TB or larger. This includes both disks being used for the Delphix Engine and other, non-related guests on the same ESX server.
The Delphix Engine may become non-responsive, may reboot unexpectedly, or may fail to reboot successfully.
Delphix Engines running 126.96.36.199, or later releases, will issue a storage fault with the following text:
Title: Critical storage device error Details: There has been an error with one or more storage devices. User Action: Contact Delphix Support for assistance.
Increase the maximum heap value for the ESX server. Delphix also recommends setting the Minimum heap value to the Maximum heap value, when possible, to guarantee that sufficient heap space will be available for the selected maximum VMDK capacity. The VMware administrator can effect these changes by configuring the VMF3.MaxHeapSizeMB and VMF3.MinHeapSizeMB variables, respectively:
- Log into the vCenter Server or ESX host using vSphere Client or VMware Infrastructure (VI) Client. When connecting to vCenter Server, select the ESX host from the inventory.
- Click the Configuration tab.
- Click Advanced Settings.
- Click VMFS3.
- Update the field in VMFS3.MaxHeapSizeMB. (see Table 1 below for sizes)
- Update the field in VMFS3.MinHeapSizeMB (optional, see Table 1 below for applicable releases)
- Reboot the ESX host for the changes to take effect
Table 1: Heap Values and Maximum Storage Sizes
|Version||Minimum heap value||Default value of maximum heap value||Maximum heap value||Default open VMDK storage per host||Maximum open VMDK storage per host|
|ESXi 5.0 Build 914586 and
|ESXi 5.0 Build 1024429 and
|ESXi 5.1 Build 914609 and
|ESXi 5.1 Build 1065491 and
For large configurations, where the total VMDK capacity would otherwise exceed 60TB, consider using virtual or physical-mapped RDM instead of VMDK disks.
See the VMware Knowledge Base article: