TB010 Delphix May Create Incomplete SnapShots After Upgrade
- Last updated
- Save as PDF
Alert Type
Data Availability
Impact
After upgrading the Delphix Engine software, some SnapShots for Oracle dSources may be incomplete. Attempts to provision a Virtual Database (VDB) from affected SnapShots will fail. This will also affect attempts to refresh or reprovision a VDB if the new SnapShot used for the VDB is impacted.
In some cases, the inability to provision or refresh may lead to a protracted outage of the affected VDB.
In some cases, the only corrective action may be to delete and re-create a dSource, which may be a time-intensive operation.
Contributing Factors
The issue may occur for appliances upgraded from a release prior to Delphix Engine 3.1.1.0 to one of the following releases:
- Delphix Engine 3.1.1.0
- Delphix Engine 3.1.2.0
- Delphix Engine 3.1.2.1
- Delphix Engine 3.1.3.0
In order for the problem to occur, SnapShots must have existed prior to the upgrade, and one or more SnapShots must have occurred since an upgrade to one of the affected Delphix Engine releases.
The issue cannot occur when the initial software version (i.e., at installation time) was Delphix Engine 3.1.1.0 or later.
Oracle 10.x databases are far more likely to be impacted than Oracle 11.x databases. Rarely, the problem might occur with Oracle 11.x databases, especially if a Delphix appliance is upgraded from a Delphix Engine release prior to 2.7.0.0, then subsequently upgraded to one of the aforementioned affected releases (3.1.1.0 - 3.1.3.0) without first completing a SnapShot.
The issue does not affect Delphix Engines that are only using SQL Server VDBs.
Even though the problem impacts snapshots created by a SnapSync operation, the problem will only be observed after a provision is attempted on an affected SnapShot.
Symptoms
When a provision job is attempted on an affected SnapShot, the provision job will fail. One or more of the following messages may occur in the Oracle Alert Log of the affected VDB:
ORA-00600: internal error code, arguments: [3020], ... ORA-10567: Redo is inconsistent with data block ...
or
Media Recovery failed with error 12801 ... ORA-1112 signalled during: alter database recover cancel...
Relief/Workaround
A dSource with an incomplete snapshot may be deleted and re-created.
If one or more SnapShots that were created prior to the upgrade exist, it's possible for Delphix Support to complete a manual procedure to recover the pre-upgrade snapshots. Then, a new SnapSync operation may be completed that will restore normal operation. Contact Delphix Support for more information.
Resolution
The issue is addressed in Delphix Engine 3.1.3.1 and later releases. Once the Delphix Engine is upgraded to 3.1.3.1 or later releases, no new incomplete snapshots will be generated. However, any existing incomplete snapshots will not be automatically repaired.
If a snapshot is completed following an upgrade to Delphix Engine 3.1.1.0, or later releases, and a VDB has been successfully provisioned from a post-upgrade SnapShot, then the parent dSource and it's SnapShots are not impacted by the issue. Future SnapShots for this dSource will also be valid.
If a VDB has not been successfully provisioned since upgrade to an affected Delphix Engine release, then it will be necessary to provision a new VDB to validate that the dSource is not impacted. In this case, start a new SnapSync operation to generate a snapshot and then provision a VDB from the new SnapShot to test.