Unable to Backup Windows Server Due to VSS Event ID 12289 (KBA1700)

Applicable Delphix Versions 

• 5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0

Issue 

Backing up a Windows host using any application that utilizes Microsoft Windows Volume Shadow Service (VSS) may fail with a VSS event ID 12289 due to the presence of large volumes (greater than 64TB in size), even if the volumes are excluded from the backup.

Backup software known to utilize Microsoft VSS and encounter this issue includes:

• Commvault
• Microsoft Windows Backup
• Veeam
• NetBackup

Prior to Delphix 5.1, Delphix only supported a maximum of 32TB for SQL Server databases. In Delphix 5.1 and higher, this was increased to support a maximum size of 255 TB for SQL Server databases. Delphix created large volumes in order to support SQL Server customers with very large databases. According to Microsoft Windows, they support volumes of up to 256TB. However, when utilizing VSS for backups, Microsoft states that the maximum size for volumes is 64TB:

If you’re using Volume Shadow Copy Service (VSS) for snapshots, the limit is 64 TB unless you’re using the VSS system provider (Volsnap) to create snapshots (instead of a hardware provider included with a SAN or RAID enclosure). In the latter case, the suggested limit for production deployments is 10 TB per volume. For more info, see "Step 2: Review the Design and Adjust as Necessary" in the Software-Defined Storage Design Considerations Guide and Usability limit for Volume Shadow Copy Service (VSS) in Windows (KB 2967756).

Further errors and information about the usability limit for VSS can be found in Microsoft KB2967766.

Troubleshooting 

Review the Windows Application log for VSS event ID 12289 similar to the following:

Log Name:      Application
Source:        VSS
Date:          7/17/2017 4:18:58 AM
Event ID:      12289
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      dctest.delphix.com
Description:
Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\Volume{ce7786a7-c83a-442d-a683-2b0d3ad0a148} - 00000000000001F4,0x0053c008,000000C513ABE800,0,000000C513ABC7E0,4096,[0]).  hr = 0x80070057, The parameter is incorrect.

The VSS event ID is often filtered up to the application log as well so check the application (Commvault, Microsoft Windows Backup, Veeam, NetBackup) logs for this error as well.

Resolution 

Delphix version 5.1.8.1 reduces the size of the volumes to 63TB so as to be compatible with VSS based backup solutions. In order to make existing virtual databases (VDBs) or dSource staging databases utilize the smaller size volumes:

1. Upgrade to Delphix 5.1.8.1 or higher.
2. Delete the dSource (and its associated VDBs) and relink the dSource from scratch.
   • If the dSource needs to be kept because of dependent VDBs, it can be detached and renamed.
   • Since VDBs are cloned from their parent, they will use the volume size the parent has. So if the dSource is still using 255T volumes, all VDBs provisioned off of it will also use 255T volumes.
3. Provision the VDB from the newly linked dSource.

The VDBs and dSources need to be recreated because there is no way to shrink the existing volumes to be under 64TB.

Additional Information 

Fixed in Delphix 5.1.8.1 via the following bug number:

DLPX-54337 - Reduce the NTFS partition size to 63T to prevent errors in Windows backups

Please contact Delphix Technical Support if your SQL Server databases would require volumes greater than 63TB.