Delphix Storage Utilization Does Not Match VMware VMDK Usage (KBA1139)
KBA
KBA#1139Problem
The nature of the Delphix Filesystem (DxFS) is such that, over time, the storage consumption reported by the Delphix Engine will not match that of VMware in the event that thin provisioning is used. This article will explore an example of this.
Applicable Delphix Versions
- Click here to view the versions of the Delphix engine to which this article applies
-
Major Release All Sub Releases 6.0 6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0 5.3
5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0, 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0 5.2
5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1
5.1
5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0
5.0
5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1, 5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4
Details
By default, VMware will thin provision VMDKs from NFS storage (v5.0 and later allow this to be overridden). Some VMware administrators also simply use thin provisioning as a standard practice in their environments; though, as a best practice; Delphix strongly recommends VMDKs be provisioned “Eager Zeroed Thick”.
As dSources and virtual databases (VDBs) change over time, DxFS will free blocks and write new blocks elsewhere on the device, due to its copy-on-write technology. However, as DxFS does not use the SCSI UNMAP operation to explicitly free these now-unused blocks (this is only tracked internally by DxFS), the hypervisor is unaware of these freed blocks. As an example, you could end up in a situation where Delphix is only using 40% of its disk resources but VMware reports 90% consumption.
To illustrate this concept, consider the simple ASCII rendering below. On the left is a DxFS perspective, where no more than 4 blocks are allocated (A) at any given time; the remaining blocks are free (F). On the right is the thin-provisioned storage perspective, as observed from the storage array and from VMware. Because DxFS does not explicitly free the blocks from the storage perspective, it considers the storage 90% “utilized”, though DxFS is only using 40%.
DxFS THIN PROVISIONED STORAGE FFFFFFFFFF FFFFFFFFFF AAFFFFFFFF AAFFFFFFFF AAAAFFFFFF AAAAFFFFFF FFAAAAFFFF AAAAAAFFFF FFAAFFAAFF AAAAAAAAFF FFFFAFAAAF AAAAAAAAAF
Below is a listing from the 'sysadmin' CLI of a Delphix engine showing the size of the storage devices assigned to the engine, along with the amount of space allocated on each device. In this example, we see that each disk has between 4.00GB and 4.48GB of space allocated.
delphix-engine storage device> list display=name,size,usedSize NAME SIZE USEDSIZE Disk2:0 70GB 21.06GB Disk2:1 8GB 4.00GB Disk2:2 8GB 4.48GB Disk2:3 8GB 4.44GB
However, exploring the underlying datastore in vSphere, the reported VMDK sizes are larger. In this example, the VMDK sizes for each virtual disk are 29.3GB, 5.8GB, 5.8GB and 5.9GB, respectively.
This discrepancy is due to the thin provisioned VMDKs having blocks allocated and freed internally by DxFS and not being reflected as a transparent action to VMware. Eventually, the size of the VMDKs will scale up to a major percentage of the provisioned size, but the capacity reported by the Delphix Engine should always be the value trusted to understand how much storage is actually being utilized, and decisions for expanding should be based on that calculation as it is what DxFS takes into account during Delphix specific processes.
This discrepancy would not be seen with thick provisioned VMDKs as their space is fully allocated within the datastore at the time of creation.