Operations on Oracle virtual databases Colocated With a Source Database may lead to data loss (ASM) (KBA1591)

Alert Type

Data Loss

Impact

In rare circumstances provisioning, refreshing, or performing a Virtual-to-Physical (V2P) operation for a Virtual Database (VDB) may result in the inadvertent deletion of data files or flashback logs in a physical Source database. In cases where data files are deleted, the affected physical database will have to be restored from backup.  If flashback logs are deleted, then the the ability  to use the optional flashback functionality on the affected physical database will be lost.  

Contributing Factors

The issue may occur in the following Delphix Releases:

• Delphix Engine 3.2.0.0
• Delphix Engine 3.2.1.0
• Delphix Engine 3.2.2.0 and 3.2.2.1
• Delphix Engine 3.2.3.0 and 3.2.3.1
• Delphix Engine 3.2.4.0, 3.2.4.1, and 3.2.4.2
• Delphix Engine 3.2.5.0 and 3.2.5.1
• Delphix Engine 3.2.6.0
• Delphix Engine 3.2.7.0 and 3.2.7.1
• Delphix Engine 4.0.0.0 and 4.0.0.1
• Delphix Engine 4.0.1.0
• Delphix Engine 4.0.2.0
• Delphix Engine 4.0.3.0
• Delphix Engine 4.0.4.0

The issue can only occur when performing operations on Oracle VDBs. Sources and VDBs using other databases are not impacted. 

The issue can only occur when operations on an Oracle VDB are targeted to the same environment (host) as the Source (physical) database. Affected operations include provisioning a new VDB, refreshing an existing VDB, or performing a Virtual-to-Physical operation (V2P) on a VDB. 

The issue can only occur when Oracle data files or flashback logs in the Source (physical) database are stored using Oracle Automated Storage Management (ASM)

The issue can only occur on physical Source databases if VDB operations for an associated Source database are attempted when the open_mode of the Source database is not "READ WRITE" or "READ ONLY".  For example, a physical Source database would be susceptible if it were shutdown or in MOUNT mode when the VDB operations occur. 

Physical Source databases that are Oracle Data Guard primaries may be susceptible to data file deletion, but they are not susceptible to flashback log deletion. Flashback is an optional Oracle feature. 

Physical Source databases that are Oracle Data Guard standby databases are not susceptible to the data file deletion issue provided there is a Managed Recovery Process (MRP) running for the Source database.  The MRP process is active as long as managed recovery is in progress.  If the open_mode of an affected Data Guard standby database is MOUNTED or "READ ONLY", its flashback logs are at risk of being deleted.  

Symptoms

One or more messages may occur in the alert log of the VDB being provisioned:

Deleted Oracle managed file +<GROUP>/<datafile>

where <GROUP> is the name of an Oracle ASM group and <datafile> is a specific datafile that is impacted. 

If flashback logs are impacted, the same message will occur except that the message will indicate the location of a flashback log instead of a datafile.  

While starting an affected Source database that has been impacted by deleted files, one or more of the following messages may be seen in the associated alert log of the physical Source database:

ORA-01157: cannot identify/lock data file <n> - see DBWR trace file
ORA-01110: data file <n>: '+<GROUP>/<datafile>
ORA-17503: ksfdopn:2 Failed to open file +<GROUP>/<datafile>
ORA-15012: ASM file '+<GROUP>/<datafile>' does not exist
Errors in file <trace file>.trc

where <n> is the "absolute file number" of an Oracle datafile, <GROUP> is the name of an Oracle ASM group, <datafile> is a specific datafile that is impacted, and <trace file> is the path of a generated Oracle trace file

When starting an affected Source database that has been impacted by deleted flashback logs, one or more of the following messages may be seen in the associated alert log of the physical Source database:

ORA-38701: Flashback database log <#> seq <#> thread <#>: "+<GROUP>/<path>"
ORA-17503: ksfdopn:2 Failed to open file +<GROUP>/<path>
ORA-15012: ASM file '+<GROUP>/<path>' does not exist

where <GROUP> is the name of the Oracle ASM group used for the flashback log and <path> is the path of the affected flashback log

Relief/Workaround

The most effective workaround for the issue is to refrain from provisioning VDBs to the same target environment (host) as a Source database. 

If there is no alternative to provisioning to the same target environment, affected operations should be limited to times when the Source databases can be guaranteed to be running and OPEN (i.e. where the open_mode of the database is either "READ WRITE" or "READ ONLY"). Care must be exercised if using policy-based VDB refreshes.

Resolution

This issue is fully resolved in Delphix release 4.0.4.1.

Additional Information

During provisioning Delphix uses Oracle to rename datafiles in the VDB's controlfile. Delphix disables Oracle's Managed Files (OMF) feature during this operation; however, because OMF is mandatory with ASM, Oracle continues to use OMF for configurations using ASM.  For the typical case when the Source database is up and running, Oracle protects datafiles with locks that will cause attempts to rename or delete OMF-managed datafiles to fail.  If the Source database isn't running, or is running in MOUNT mode, these locks will not be active, and datafiles and flashback logs may be inadvertently deleted.