Upgrading Oracle dSources and Virtual Databases and the Implications for Self Service Container Refresh Operations (KBA7157)

Capture a backup of the dSource database

In this scenario RMAN is used to capture a copy of the database prior to upgrade.  This was captured to ensure a fallback position is available above and beyond the restore point set during the upgrade of the database itself.

Recovery Manager: Release 12.2.0.1.0 - Production on Wed Mar 10 16:49:55 2021

Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.
connected to target database: DB1221 (DBID=1325688878)

RMAN> backup database include current controlfile format '/u01/app/oracle/backups/fulld_db_%s_%p.bck';

Starting backup at 10-MAR-21
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=51 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00003 name=+DATA/DB1221/DATAFILE/sysaux.281.1064253477
input datafile file number=00001 name=+DATA/DB1221/DATAFILE/system.280.1064253433
input datafile file number=00004 name=+DATA/DB1221/DATAFILE/undotbs1.282.1064253503
input datafile file number=00007 name=+DATA/DB1221/DATAFILE/users.283.1064253503
channel ORA_DISK_1: starting piece 1 at 10-MAR-21
channel ORA_DISK_1: finished piece 1 at 10-MAR-21
piece handle=/u01/app/oracle/backups/fulld_db_94_1.bck tag=TAG20210310T165052 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:45
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00010 name=+DATA/DB1221/BB0B2FAE5D4E4929E0550A00270A81F8/DATAFILE/sysaux.300.1064254113
input datafile file number=00009 name=+DATA/DB1221/BB0B2FAE5D4E4929E0550A00270A81F8/DATAFILE/system.299.1064254113
input datafile file number=00011 name=+DATA/DB1221/BB0B2FAE5D4E4929E0550A00270A81F8/DATAFILE/undotbs1.298.1064254113
input datafile file number=00012 name=+DATA/DB1221/BB0B2FAE5D4E4929E0550A00270A81F8/DATAFILE/users.302.1064254147
channel ORA_DISK_1: starting piece 1 at 10-MAR-21
channel ORA_DISK_1: finished piece 1 at 10-MAR-21
piece handle=/u01/app/oracle/backups/fulld_db_95_1.bck tag=TAG20210310T165052 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00006 name=+DATA/DB1221/4700A987085B3DFAE05387E5E50A8C7B/DATAFILE/sysaux.293.1064253697
input datafile file number=00005 name=+DATA/DB1221/4700A987085B3DFAE05387E5E50A8C7B/DATAFILE/system.294.1064253697
input datafile file number=00008 name=+DATA/DB1221/4700A987085B3DFAE05387E5E50A8C7B/DATAFILE/undotbs1.295.1064253697
channel ORA_DISK_1: starting piece 1 at 10-MAR-21
channel ORA_DISK_1: finished piece 1 at 10-MAR-21
piece handle=/u01/app/oracle/backups/fulld_db_96_1.bck tag=TAG20210310T165052 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 10-MAR-21
channel ORA_DISK_1: finished piece 1 at 10-MAR-21
piece handle=/u01/app/oracle/backups/fulld_db_97_1.bck tag=TAG20210310T165052 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 10-MAR-21


Starting Control File and SPFILE Autobackup at 10-MAR-21
piece handle=+DATA/DB1221/AUTOBACKUP/2021_03_10/s_1066841531.442.1066841531 comment=NONE
Finished Control File and SPFILE Autobackup at 10-MAR-21

A backup of the archivelogs as they existed prior to the upgrade was also captured.

RMAN> backup archivelog all delete input format '/u01/app/oracle/backups/arch_all_%s_%p.bck';

Starting backup at 10-MAR-21
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=54 RECID=87 STAMP=1065439833
input archived log thread=1 sequence=55 RECID=88 STAMP=1065439867
input archived log thread=1 sequence=56 RECID=89 STAMP=1065439868
input archived log thread=1 sequence=57 RECID=90 STAMP=1065439899
input archived log thread=1 sequence=58 RECID=91 STAMP=1065439919
input archived log thread=1 sequence=59 RECID=92 STAMP=1065439920
..
input archived log thread=1 sequence=111 RECID=170 STAMP=1066456850
input archived log thread=1 sequence=112 RECID=171 STAMP=1066475244
channel ORA_DISK_1: starting piece 1 at 10-MAR-21
channel ORA_DISK_1: finished piece 1 at 10-MAR-21
piece handle=/u01/app/oracle/backups/arch_all_99_1.bck tag=TAG20210310T165327 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:01:55
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=6 RECID=2 STAMP=1064343826
input archived log thread=1 sequence=7 RECID=3 STAMP=1064344144
input archived log thread=1 sequence=8 RECID=4 STAMP=1064379642
..
input archived log thread=1 sequence=52 RECID=85 STAMP=1065357739
input archived log thread=1 sequence=53 RECID=86 STAMP=1065399753
channel ORA_DISK_1: starting piece 1 at 10-MAR-21
channel ORA_DISK_1: finished piece 1 at 10-MAR-21
piece handle=/u01/app/oracle/backups/arch_all_101_1.bck tag=TAG20210310T165327 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:02:05
channel ORA_DISK_1: deleting archived log(s)
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_02_12/thread_1_seq_6.305.1064343827 RECID=2 STAMP=1064343826
..
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_06/thread_1_seq_111.417.1066456847 RECID=170 STAMP=1066456850
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_06/thread_1_seq_112.418.1066475239 RECID=171 STAMP=1066475244
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=113 RECID=172 STAMP=1066495708
input archived log thread=1 sequence=114 RECID=173 STAMP=1066513152
..
input archived log thread=1 sequence=135 RECID=222 STAMP=1066837204
input archived log thread=1 sequence=136 RECID=225 STAMP=1066841607
channel ORA_DISK_1: starting piece 1 at 10-MAR-21
channel ORA_DISK_1: finished piece 1 at 10-MAR-21
piece handle=/u01/app/oracle/backups/arch_all_102_1.bck tag=TAG20210310T165327 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:01:45
channel ORA_DISK_1: deleting archived log(s)
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_06/thread_1_seq_113.419.1066495703 RECID=172 STAMP=1066495708
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_06/thread_1_seq_114.420.1066513147 RECID=173 STAMP=1066513152
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_07/thread_1_seq_115.421.1066523365 RECID=174 STAMP=1066523371
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_07/thread_1_seq_116.422.1066547415 RECID=175 STAMP=1066547420
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_07/thread_1_seq_117.423.1066561809 RECID=176 STAMP=1066561815
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_07/thread_1_seq_118.424.1066582243 RECID=177 STAMP=1066582248
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_07/thread_1_seq_119.425.1066599985 RECID=178 STAMP=1066599990
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_08/thread_1_seq_120.426.1066608079 RECID=179 STAMP=1066608083
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_08/thread_1_seq_121.427.1066637501 RECID=180 STAMP=1066637505
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_08/thread_1_seq_122.428.1066687537 RECID=181 STAMP=1066687542
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_09/thread_1_seq_123.429.1066730415 RECID=182 STAMP=1066730420
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_09/thread_1_seq_124.430.1066777499 RECID=183 STAMP=1066777505
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_10/thread_1_seq_125.431.1066827037 RECID=184 STAMP=1066827039
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_10/thread_1_seq_126.432.1066827039 RECID=185 STAMP=1066827039
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_10/thread_1_seq_127.433.1066834433 RECID=186 STAMP=1066834432
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_10/thread_1_seq_128.434.1066834443 RECID=189 STAMP=1066834444
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_10/thread_1_seq_129.435.1066836529 RECID=210 STAMP=1066836529
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_10/thread_1_seq_130.436.1066836617 RECID=212 STAMP=1066836616
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_10/thread_1_seq_131.437.1066836617 RECID=214 STAMP=1066836617
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_10/thread_1_seq_132.438.1066836647 RECID=216 STAMP=1066836647
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_10/thread_1_seq_133.439.1066836677 RECID=218 STAMP=1066836677
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_10/thread_1_seq_134.440.1066836677 RECID=220 STAMP=1066836677
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_10/thread_1_seq_135.441.1066837205 RECID=222 STAMP=1066837204
archived log file name=+DATA/DB1221/ARCHIVELOG/2021_03_10/thread_1_seq_136.443.1066841607 RECID=225 STAMP=1066841607
Finished backup at 10-MAR-21


Starting Control File and SPFILE Autobackup at 10-MAR-21
piece handle=+DATA/DB1221/AUTOBACKUP/2021_03_10/s_1066841971.443.1066841973 comment=NONE
Finished Control File and SPFILE Autobackup at 10-MAR-21

Perform the primary site (and dSource) Oracle Upgrade

This upgrade of the source database is performed using the Oracle Database Upgrade Assistant, dbua.  The upgrade assistant is started from the Oracle Home where the higher version is in place and the one to which the upgrade is going to moving the dSource database.

Working through the dbua steps.

1. Select the source Container database.
   
    
2. Select the pluggable database within the container to upgrade.
   
    
3. Have Oracle perform its pre-upgrade validation checks.
    
4. Choose how the upgrade is to be performed.
    
5. Let dbua know how you want the upgrade to handle how it would be rolled back should this be required.  A backup was taken prior to beginning this process so using a guaranteed restore point was picked in this upgrade.
    
6. Let Oracle know how the listener is to be managed post-upgrade.
    
7. Enterprise manager is not used for this database so this is deselected as a part of this upgrade.
    
8. Check the summary screen to ensure that the upgrade is going to be performed according to the requirements of the DBA.
    
9. Initiate the upgrade. This will take a while.
   
    
10. Check the upgrade results to ensure it successfully completed.
    
     
11. Post-upgrade the dSource compatible parameter is left untouched and shows 12.2.0.

SQL> show parameter compat

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
compatible                           string      12.2.0

Upgrade the standby site

As a Data Guard configuration and physical standby database is in place, this is also upgraded after the primary site has been upgraded.  While not strictly a part of the Delphix configuration it is included in this knowledge base article in case it has been linked as a dSource, as is the case for many Delphix customers.

1. Copy the standby site parameter file and password file to the new Oracle Home.

[oracle@oel7sitde2 ~]$ cp /u01/app/oracle/product/12.2.0/dbhome_1/dbs/spfiledb1221.ora /u01/app/oracle/19.0.0/dbhome_1/dbs/
[oracle@oel7sitde2 ~]$ cp /u01/app/oracle/product/12.2.0/dbhome_1/dbs/orapwdb1221 /u01/app/oracle/19.0.0/dbhome_1/dbs/
[oracle@oel7sitde2 ~]$ ls -l /u01/app/oracle/19.0.0/dbhome_1/dbs/
total 18336
-rw-r-----. 1 oracle asmadmin     8192 Jan 21 21:03 dr1cdb19c1.dat
-rw-r-----. 1 oracle asmadmin     8192 Jan 21 21:03 dr2cdb19c1.dat
-rw-rw----. 1 oracle asmadmin     1544 Feb 11 11:38 hc_cdb19c1.dat
-rw-r--r--. 1 oracle oinstall     3079 May 14  2015 init.ora
-rw-r-----. 1 oracle asmadmin       24 Jan 14 22:55 lkCDB19C1
-rw-r-----. 1 oracle oinstall     2048 Jan 14 22:57 orapwcdb19c1
-rw-r-----. 1 oracle oinstall     3584 Mar 12 09:46 orapwdb1221
-rw-r-----. 1 oracle oinstall     2048 Jan 29 18:43 orapwVCDB19C1
-rw-r-----. 1 oracle asmadmin 18726912 Jan 21 21:31 snapcf_cdb19c1.f
-rw-r-----. 1 oracle oinstall     5632 Mar 12 09:45 spfiledb1221.ora

2. Shut the standby database from the 12.2.0.1 RDBMS Home.

[oracle@oel7sitde2 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Fri Mar 12 09:46:55 2021

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> shutdown immediate
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

3. Start the database up from the Oracle 19c RDBMS Home.

[oracle@oel7sitde2 ~]$ cat /etc/oratab 
#Backup file is  /u01/app/orabase/crsdata/oel7sitde2/output/oratab.bak.oel7sitde2.grid line added by Agent
#
# This file is used by ORACLE utilities.  It is created by root.sh
# and updated by either Database Configuration Assistant while creating
# a database or ASM Configuration Assistant while creating ASM instance.
..
dbhome1:/u01/app/oracle/19.0.0/dbhome_1:N
cdb19c1:/u01/app/oracle/19.0.0/dbhome_1:N # line added by Agent
#db1221:/u01/app/oracle/product/12.2.0/dbhome_1:N # line added by Agent
db1221:/u01/app/oracle/19.0.0/dbhome_1:N # line added by Agent
+ASM:/u01/app/grid/19.0.0:N # line added by Agent

[oracle@oel7sitde2 ~]$ . oraenv
ORACLE_SID = [db1221] ? 
The Oracle base has been changed from /u01/app/oracle to /u01/app/orabase

[oracle@oel7sitde2 ~]$ which sqlplus 
/u01/app/oracle/19.0.0/dbhome_1/bin/sqlplus

[oracle@oel7sitde2 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Mar 12 09:50:32 2021
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Connected to an idle instance.

SQL> startup mount
ORACLE instance started.

Total System Global Area 1174405120 bytes
Fixed Size     9134080 bytes
Variable Size   318767104 bytes
Database Buffers   838860800 bytes
Redo Buffers     7643136 bytes
Database mounted.

4. Through managed recovery the physical standby database will synchronize itself with the primary site and the physical standby RDBMS itself will be upgraded through redo being applied to it.

[oracle@oel7sitde2 ~]$ tail -f /u01/app/oracle/diag/diag/rdbms/db1221sb/db1221/trace/alert_db1221.log 
..
2021-03-12T10:13:12.688595+11:00
PR00 (PID:11894): Media Recovery Log +FRA/DB1221SB/ARCHIVELOG/2021_03_12/thread_1_seq_179.415.1066990249

2021-03-12T10:13:26.379028+11:00
PR00 (PID:11894): Media Recovery Log +FRA/DB1221SB/ARCHIVELOG/2021_03_12/thread_1_seq_180.364.1066990269

2021-03-12T10:13:36.828242+11:00
PR00 (PID:11894): Media Recovery Log +FRA/DB1221SB/ARCHIVELOG/2021_03_12/thread_1_seq_181.414.1066990249

5. The standby instance is running from the 19c home and has been upgraded to 19c.

SQL> select instance_name,version,version_full,edition from v$instance;

INSTANCE_NAME    VERSION           VERSION_FULL      EDITION
---------------- ----------------- ----------------- -------
db1221           19.0.0.0.0        19.3.0.0.0        EE

6. Compatible is however left set to 12.2.0 in line with the primary site in the Data Guard configuration.

SQL> show parameter compatible

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
compatible                           string      12.2.0

If no change to the Delphix configuration is made to indicate to Delphix the upgrade is completed, attempts to snapshot the dSource will fail reporting the error seen in the following screen shot.  This needs to be corrected by following the dSource upgrade steps here. 

Upgrade the dSource configuration on the Delphix side

Delphix changes are required to let Delphix know the Oracle upgrade has occurred and the VDB is now to run from a new and higher release Oracle Home.  This involves pointing Delphix to the new Oracle Home for the dSource.  Complete this process to make the appropriate alterations within Delphix, no changes are made to the source database itself or its configuration from an Oracle perspective. 

The upgrade workflow for the dSource is used to perform this task.

Note:

This new Oracle home will need to have been discovered by Delphix through environment refresh after the new install is laid down on the dSource host environment.

1. Select the new Oracle Home and version to associate with the database. 

Both the dSource PDB and CDB should show the new RDBMS Home in place in the Delphix GUI (and CLI) once this step has been completed.

2. The post upgrade snapshot of the dSource can now be captured successfully.

 

Demonstrating a failed Self Service refresh of a container holding a VDB whose parent dSource has been upgraded.

An example of an attempt to perform the Self Service refresh without altering the VDB configuration associated with the Self Service container is shown below.

Upgrade the Virtual Database associated with the Self Service Container. 

Uprading the VDB from a Delphix perspective involves pointing the VDB to what will be its new Oracle Home.  In this case an Oracle 19c Enterprise Edition Oracle Home.

 

At this point if a refresh is attempted it will fail with a different error to the one reporting the database requires an upgrade that was seen previously.  The initial VDB snapshot(s), taken as a part of the Self Service refresh and created just in case an automatic rollback of the refresh may be required, are now failing. The VDB settings for the Oracle Home in Delphix are now 19c while the VDB instance itself is still running from the 12.2 Oracle Home.  This version mismatch is now reflected in the error and reports the version differences.

Given this combination of errors, it looks like a refresh cannot be performed because Self Service requires one version of the database be set in Delphix at the start of the refresh to capture the snapshot and another release required to complete the refresh itself.

The following behaviors are encountered if the Delphix GUI is used to perform the refresh:

• Not upgrading the VDB in Delphix results in Delphix attempting to start the refreshed VDB from the old 12.2 home when the snapshot has been captured from the 19c home.
• Upgrading the VDB in Delphix results in the initial Self Service snapshot failing as Delphix is detecting the version held by Delphix being different to that of the running instance.

Performing the Self Service refresh successfully.

A Self Service refresh can be performed successfully however, currently, only through using the Delphix Command Line Interface and enabling the "force" option through the CLI.  The Delphix GUI cannot be used for this task.  

 

Warning:

The force option in the CLI will bypass the initial snapshot Self Service would normally capture and introduces additional risk into performing the refresh operation.  Should the refresh fail for some reason and Delphix is unable to create the new version of the VDB from the dSource snapshot the refresh has used, Self Service no longer has its fallback position available, where it would normally revert the Self Service container to in cases of a refresh failure.  This can leave the Self Service container in a state where it may be unrecoverable.

The following conditions must be met for the refresh to complete:

• The dSource upgraded to 19c and a refresh is to be performed using a snapshot captured after the move to 19c.
• The VDB to be refreshed is currently running in 12.2.
• The VDB upgraded to 19c in Delphix.
• The force option set for the Self Service Container refresh in the CLI.
• In this example the compatible parameter remains consistent across the releases.

Using the CLI to refresh the Self Service Container object.

dlpx6050.plb.internal> selfservice container

1. Select the container where the refresh is to be performed.

dlpx6050.plb.internal selfservice container> ls
Objects
NAME                              TEMPLATE                         NOTES                             ACTIVEBRANCH
Oracle_Upgrade_Testing_Container  Oracle_Upgrade_Testing_Template  Oracle Upgrade Testing Container  default

..
dlpx6050.plb.internal selfservice container> select Oracle_Upgrade_Testing_Container

2. Set the refresh option to force the refresh to bypass the initial snapshot operations Self Service would normally perform.

dlpx6050.plb.internal selfservice container 'Oracle_Upgrade_Testing_Container'> refresh
dlpx6050.plb.internal selfservice container 'Oracle_Upgrade_Testing_Container' refresh *> ls
Properties
    type: JSDataContainerRefreshParameters
    forceOption: false
dlpx6050.plb.internal selfservice container 'Oracle_Upgrade_Testing_Container' refresh *> 
dlpx6050.plb.internal selfservice container 'Oracle_Upgrade_Testing_Container' refresh *> set forceOption=true
dlpx6050.plb.internal selfservice container 'Oracle_Upgrade_Testing_Container' refresh *> ls
Properties
    type: JSDataContainerRefreshParameters
    forceOption: true (*)

3. Initiate the refresh using the CLI.

dlpx6050.plb.internal selfservice container 'Oracle_Upgrade_Testing_Container' refresh *> commit
    Dispatched job JOB-65
    SELFSERVICE_USER_CONTAINER_REFRESH job started for "Oracle_Upgrade_Testing_Container".
    SELFSERVICE_USER_CONTAINER_REFRESH job for "Oracle_Upgrade_Testing_Container" completed successfully.

 

 

---