Provisioning VDBs to a SQL Server failover cluster in Azure fails due to BitLocker (KBA10507)
KBA
KBA#Applicable Delphix Versions
- Click here to view the versions of the Delphix engine to which this article applies
-
Date Release All All
Troubleshooting VDB Provision Failure due to BitLocker
Provisioning a VDB to a failover cluster in Azure fails because BitLocker is enabled by default. In other platforms, BitLocker is not enabled by default allowing Delphix to successfully add the VDB's disk to the cluster.
When provisioning to a failover cluster in Azure, Delphix will fail to add a disk and receive the following error:
TIMER(0.565): Add cluster disk
ERROR_GET_CLUSTER_DISK_RESOURCE
[2023-08-28T15:43:55,536][DEBUG][host.WindowsHostUtils#dumpStdoutAndStderr:93][Worker-739141|JOB-447030|DB_PROVISION(MSSQL_DB_CONTAINER-926)][ACTION-7228649] stderr from E:\dlpx_cluster_vdbs\f657c417-7449-eb40-8acf-10fcf2800b91-vdb-588\SCRIPT\MountLunData.ps1
Add-ClusterDisk : An error was encountered while creating storage resources for 'CLUSTER_NAME'.
The resource 'Cluster Disk 1' did not come online.
The desired state change for 'Cluster Disk 1' did not occur before the timeout expired.
Use the PowerShell Get-ClusterLog command to collect logs to show the cause of the failure. The following command will collect logs from all cluster nodes, and place them in the current directory of the node from which it is run, allowing you to easily search each log file:
Get-ClusterLog -Destination .
Systems experiencing this issue will show failures due to a BitLocker error Failed during checks for Bitlocker enabled volumes. Error 21 in one or more of the generated log files.
[System] 00001c34.00009c5c::2023/08/28-19:43:55.149 ERR Cluster physical disk resource online failed.
Physical Disk resource name: Cluster Disk 1
Device Number: 14
Device Guid: {d35b7713-510f-e276-4f42-05b2bc980d18}
Error Code: 21
Additional reason: CheckUnlockBitlockerFailure
[System] 00000d14.0000b9a8::2023/08/28-19:43:55.158 ERR Cluster resource 'Cluster Disk 1' of type 'Physical Disk' in clustered role 'Available Storage' failed. The error code was '0x15' ('The device is not ready.').
...etc...
[Verbose] 00001c34.00009c5c::2023/08/28-19:43:55.149 INFO [RES] Physical Disk <Cluster Disk 1>: UnLockVolumesIfEncryptionEnabled
[Verbose] 00001c34.00009c5c::2023/08/28-19:43:55.149 ERR [RES] Physical Disk <Cluster Disk 1>: FveOpenVolumeProcAddr Failed status 0x80070015
[Verbose] 00001c34.00009c5c::2023/08/28-19:43:55.149 ERR [RES] Physical Disk <Cluster Disk 1>: UnLockVolumesIfEncryptionEnabled Failed with 80070015
[Verbose] 00001c34.00009c5c::2023/08/28-19:43:55.149 ERR [RES] Physical Disk <Cluster Disk 1>: OnlineThread: Failed during checks for Bitlocker enabled volumes. Error 21
The error CheckUnlockBitlockerFailure may also be visible in the Windows System Event log on one or more of the servers in the cluster.
Prerequisites
This issue has been confirmed to happen in Azure. Other platforms (on-premises, AWS, etc.) do not appear to be impacted in their default configurations because BitLocker is not enabled. However, this issue can be reproduced on other platforms by enabling BitLocker.
Resolution
To resolve the Failed during checks for Bitlocker enabled volumes. Error 21 error, it is necessary to uninstall BitLocker from all nodes in the cluster. Disabling Bitlocker is not sufficient to resolve the issue.
Microsoft Technical Support noted:
We removed the Bitlocker encryption feature from Server Manager for both the nodes by draining and pausing the nodes sequentially. Once this step was completed, we were able to successfully add Disk 7 in the cluster both manually and using your automated process.
Hence, we would recommend you to remove the BitLocker Feature from the Cluster Nodes, if in future you are facing a similar issue for the Azure Environment, as BitLocker feature is installed by default in Azure Windows Virtual Machines.
The following scenarios are noted:
- Provisioning works with BitLocker not installed on Node 1 and BitLocker Installed on Node 2 if Node 1 is running the "Available Storage" and "SQL Server" Cluster Groups. Failover attempts will throw a
CheckUnlockBitlockerFailureerror on Node 2. - Provisioning fails with BitLocker installed on both nodes, reporting
CheckUnlockBitlockerFailureon whichever server is hosting the "Available Storage" Cluster Group. - Provisioning works again when both nodes have BitLocker uninstalled.