After upgrading the Delphix Engine software, some SnapShots for Oracle dSources may be incomplete. Attempts to provision a Virtual Database (VDB) from affected SnapShots will fail. This will also affect attempts to refresh or reprovision a VDB if the new SnapShot used for the VDB is impacted.
In some cases, the inability to provision or refresh may lead to a protracted outage of the affected VDB.
In some cases, the only corrective action may be to delete and re-create a dSource, which may be a time-intensive operation.
The issue may occur for appliances upgraded from a release prior to Delphix Engine 18.104.22.168 to one of the following releases:
- Delphix Engine 22.214.171.124
- Delphix Engine 126.96.36.199
- Delphix Engine 188.8.131.52
- Delphix Engine 184.108.40.206
In order for the problem to occur, SnapShots must have existed prior to the upgrade, and one or more SnapShots must have occurred since an upgrade to one of the affected Delphix Engine releases.
The issue cannot occur when the initial software version (i.e., at installation time) was Delphix Engine 220.127.116.11 or later.
Oracle 10.x databases are far more likely to be impacted than Oracle 11.x databases. Rarely, the problem might occur with Oracle 11.x databases, especially if a Delphix appliance is upgraded from a Delphix Engine release prior to 18.104.22.168, then subsequently upgraded to one of the aforementioned affected releases (22.214.171.124 - 126.96.36.199) without first completing a SnapShot.
The issue does not affect Delphix Engines that are only using SQL Server VDBs.
Even though the problem impacts snapshots created by a SnapSync operation, the problem will only be observed after a provision is attempted on an affected SnapShot.
When a provision job is attempted on an affected SnapShot, the provision job will fail. One or more of the following messages may occur in the Oracle Alert Log of the affected VDB:
ORA-00600: internal error code, arguments: , ... ORA-10567: Redo is inconsistent with data block ...
Media Recovery failed with error 12801 ... ORA-1112 signalled during: alter database recover cancel...
A dSource with an incomplete snapshot may be deleted and re-created.
If one or more SnapShots that were created prior to the upgrade exist, it's possible for Delphix Support to complete a manual procedure to recover the pre-upgrade snapshots. Then, a new SnapSync operation may be completed that will restore normal operation. Contact Delphix Support for more information.
The issue is addressed in Delphix Engine 188.8.131.52 and later releases. Once the Delphix Engine is upgraded to 184.108.40.206 or later releases, no new incomplete snapshots will be generated. However, any existing incomplete snapshots will not be automatically repaired.
If a snapshot is completed following an upgrade to Delphix Engine 220.127.116.11, or later releases, and a VDB has been successfully provisioned from a post-upgrade SnapShot, then the parent dSource and it's SnapShots are not impacted by the issue. Future SnapShots for this dSource will also be valid.
If a VDB has not been successfully provisioned since upgrade to an affected Delphix Engine release, then it will be necessary to provision a new VDB to validate that the dSource is not impacted. In this case, start a new SnapSync operation to generate a snapshot and then provision a VDB from the new SnapShot to test.