Skip to main content
Delphix

Unable to backup Windows server due to VSS Event ID 12289

Applicable Delphix Versions

  • 5.1

Issue

Backing up a Windows host using any application that utilizes Microsoft Windows Volume Shadow Service (VSS) may fail with a VSS event ID 12289 due to the presence of large volumes (greater than 64TB in size), even if the volumes are excluded from the backup.

Backup software known to utilize Microsoft VSS and encounter this issue includes:

  • Commvault
  • Microsoft Windows Backup
  • Veeam
  • NetBackup

Prior to Delphix 5.1, Delphix only supported a maximum of 32TB for SQL Server databases. In Delphix 5.1 and higher, this was increased to support a maximum size of 255 TB for SQL Server databases. Delphix created large volumes in order to support SQL Server customers with very large databases. According to Microsoft Windows, they support volumes of up to 256TB. However, when utilizing VSS for backups, Microsoft states that the maximum size for volumes is 64TB:

If you’re using Volume Shadow Copy Service (VSS) for snapshots, the limit is 64 TB unless you’re using the VSS system provider (Volsnap) to create snapshots (instead of a hardware provider included with a SAN or RAID enclosure). In the latter case, the suggested limit for production deployments is 10 TB per volume. For more info, see "Step 2: Review the Design and Adjust as Necessary" in the Software-Defined Storage Design Considerations Guide and Usability limit for Volume Shadow Copy Service (VSS) in Windows (KB 2967756).

Further errors and information about the usability limit for VSS can be found in Microsoft KB2967766.

Troubleshooting

Review the Windows Application log for VSS event ID 12289 similar to the following:

Log Name:      Application
Source:        VSS
Date:          7/17/2017 4:18:58 AM
Event ID:      12289
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      dcqwdbs468.root.sutterhealth.org
Description:
Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\Volume{ce7786a7-c83a-442d-a683-2b0d3ad0a148} - 00000000000001F4,0x0053c008,000000C513ABE800,0,000000C513ABC7E0,4096,[0]).  hr = 0x80070057, The parameter is incorrect.

The VSS event ID is often filtered up to the application log as well so check the application (Commvault, Microsoft Windows Backup, Veeam, NetBackup) logs for this error as well.

Resolution

Delphix version 5.1.8.1 reduces the size of the volumes to 63TB so as to be compatible with VSS based backup solutions. In order to make virtual databases (VDBs) or dSource staging databases utilize the smaller size volumes:

  1. Upgrade to Delphix 5.1.8.1.
  2. Delete or refresh the VDB.
  3. Delete the dSource (and its associated VDBs) and relink the dSource from scratch.

The VDBs and dSources need to be recreated because there is no way to shrink the existing volumes to be under 64TB.

Additional Information

Fixed in Delphix 5.1.8.1 via the following bug number:

DLPX-54337 - Reduce the NTFS partition size to 63T to prevent errors in Windows backups

Please contact Delphix Technical Support if your SQL Server databases would require volumes greater than 63TB.