Restarting Validated Sync when old backup files are no longer present (KBA7453)

Issue

If Validated Sync for a SQL Server dSource is stopped for an extended period of time and then restarts, the following Fault may be reported:

Title: Backup file for the source database was not found in the backup locations

Target: MySourceDB

Details: For dSource "MySourceDB", backup file "MySourceDB_backup_2021_01_01.bak" was not found in the backup locations [\\FILESERVER\ShareName\Backups].

User Action: Make sure the backup locations are correct and the user "Delphix_OS" has the required read privileges on them.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
6.0	6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0
5.3	5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0, 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0
5.2	5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1
5.1	5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0
5.0	5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1, 5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

Resolution

As a first step, verify that the current backup location is correct, and that the backup does not exist.

If the backup file does exist in the specified location, ensure that the user reported in the fault has permissions to access the backup file.

If the backup location is correct, and the backup file no longer exists or is not available, there are two ways to resolve this issue:

Make the requested backup file available, if possible
Use the Snapshot function to force the Delphix Engine to skip any backups that are no longer available

If a gap in the Timeflow can be tolerated, the fastest resolution is to use the Snapshot function, with the Use the most recent full or differential backup option selected.

Making the requested backup file available

By making any missing backup files available, the Delphix Engine will be able to continue processing all snapshots without creating a gap in the Timeflow.

If the backup file mentioned in the Fault is available in another location or in a long-term backup archive, retrieve the missing backup(s) and make them available to the Delphix Engine.

To help the Delphix Engine locate these files, it may be necessary to temporarily configure an additional Backup Path for the dSource. For more information on configuring Backup Paths, see our document Data Management Settings for SQL Server Data Sources.

Skipping backups using the Snapshot function

Using the Snapshot feature will allow the Validated Sync process to skip any missing backups. This will create a gap in the Timeflow, which will be visible until the snapshots are cleaned up by Retention.

To manually trigger a snapshot, open the Manage → Datasets screen, and then click the camera () icon to open the Snapshot dialog.

This will provide three options:

Taking a copy-only full backup will initiate a new full backup of the Source Database to a temporary backup file using Delphix Engine storage. Due to the additional load that this will place on the Source and Staging databases, this is not recommended if a backup file is already available.
Using the most recent full or differential backup will find the most recent available matching backup. This will create a gap between the current dSource snapshot and the latest available Source Backup and is the recommended option if this gap in the Timeflow is acceptable.
Using a specific full or differential backup will allow you to specify the Backup Set UUID (identified in the the msdb.dbo.backupset table) of the earliest backup file that is still available. This will create a gap between the current dSource snapshot and the timestamp of that backup file, and Validated Sync will start to "catch up" once this snapshot is completed.

Once the Snapshot is completed, the Validated Sync process will automatically start monitoring for new Source Database backups. The Validated Sync Fault will automatically clear once this process is restarted.

Analysis

The SQL Server Validated Sync process is designed to maintain a continuous Timeflow, retrieving and applying all backups of the configured type (Full, Full+Diff, or Transaction Log).

To make sure that you always have access to the expected snapshots, it will not automatically "skip" old backups, and will raise a Fault until the backup file is made available or a manual snapshot is taken.

There are several reasons that Validated Sync may stop for an extended period, including:

The dSource or Staging Database has been explicitly disabled
The Delphix Engine reached its capacity threshold, and has paused Validated Sync, as described in An Overview of Capacity and Performance Information
There are problems on the Staging Environment, such as connectivity loss or disruption to the Delphix Connector service
A backup has been taken to a location which is not accessible to the Staging Environment
The backup location has changed, and the Staging database does not have connectivity or permissions to this location
The Staging Environment or Delphix Engine has lost connectivity to the Source Database or backup file share

In other cases, Validated Sync may not be able to create snapshots as quickly as new backups are being generated.

If this issue relates to an unexpected failure, the Delphix Engine will raise a Fault or Alert when that failure occurs and will also send SNMP or email notifications, if configured to do so.

KBA