Resolving "Restore of source database backup failed" Faults when using Commvault Ingestion (KBA10405)

Issue

When using Commvault Ingestion with SQL Server dSources, the following fault may be intermittently raised:

Fault code:
fault.mssql.stagingdb.restore.failed.transient

Title:
Restore of source database backup failed

Description:
Restore of backup with UUID "AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA" for source "SourceDatabase" failed on the staging host "myhost.mydomain.com": Commvault restore job with job id '123456' failed with job status 'Failed' and following reason: There is nothing to restore, database [SourceDatabase] is already up to date.

Action:
Make sure that the backup is not corrupt, the backup locations for the dSource [SourceDatabase] are correct, the SQL Server instance "MSSQLSERVER" on the staging host is up and that user "mydomain\myuser" has privileges to restore a database on the host. If the problem does not resolve, perform a sync on the dSource to restore a full database backup. If the problem persists, contact Delphix support.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Date	Release
Sep 21, 2023	15.0.0.0
Aug 24, 2023	14.0.0.0
Jul 24, 2023	13.0.0.0
Jun 21, 2023	12.0.0.0
May 25, 2023	11.0.0.0
Apr 13, 2023	10.0.0.0 | 10.0.0.1
Mar 13, 2023 | Mar 20, 2023	9.0.0.0 | 9.0.0.1
Feb 13, 2023	8.0.0.0
Jan 12, 2023	7.0.0.0
Releases Prior to 2023
Major Release	All Sub Releases
6.0	6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0, 6.0.13.1, 6.0.14.0, 6.0.15.0, 6.0.16.0, 6.0.17.0, 6.0.17.1, 6.0.17.2
5.3	5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0, 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0
5.2	5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1
5.1	5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0
5.0	5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1, 5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

Cause

This fault may be caused by a timing mismatch in how each product interacts with SQL Server backup metadata:

• Commvault processes each database in a Subclient as part of a single backup Job. It does not make the backups available for restore until the Job is completed.
• SQL Server records each successful backup to the msdb database immediately after each backup is complete. It does not record whether the Commvault Job is still in progress.
• The Continuous Data Engine regularly monitors the msdb database for new backup records. As part of the Validated Sync process, it will attempt to initiate a Commvault restore job within one minute of a new backup being detected.

As a result of the above behaviors, the Continuous Data Engine may attempt to initiate a Commvault Restore Job before the backup is fully available in Commvault. If this happens, the Job will fail with the following error, triggering a Fault within the Continuous Data Engine.

There is nothing to restore, database [SourceDatabase] is already up to date.

Note:

The Continuous Data Engine will attempt to retry the Commvault restore operation at regular intervals. Once the Commvault Job is complete and the backup is available for restore, the next Validated Sync attempt will succeed, and this fault will clear automatically.

Resolution

If this error does not occur regularly, and the Fault does not disrupt your administration of the Continuous Data Engine, you may disregard the Fault. It will clear automatically once the backup is available for restore by the Validated Sync process of the dSource.

If this is not possible, two resolutions are currently available: