Oracle Snapsync Fails Reporting the Selected Instances Are Not in the Databases Configuration (KBA9457)

Issue

Attempts to take a snapshot of a source Oracle database are failing reporting an error message like the following:

Run SnapSync for database "cdb1911a".

Error
Delphix Engine cannot connect to any host to load data from source "cdb1911a": Selected instances "[instanceNumber: 3 instanceName: cdb1911a3: hostName: oelc5n1.plb.internal]" are not in database "cdb1911a" configuration.

Error Code
exception.oracle.dsource.sync.no_hosts.rac

Suggested Action
Refresh host to discover database instances, or add database configuration manually. If the database was added manually, make sure that the database name and database unique name provided in Delphix match Oracle exactly.

[oracle@oelc5n3 ~]$ srvctl config database -d cdb1911a

Database unique name: cdb1911a
Database name: cdb1911a
Oracle home: /u01/app/oracle/19.11.1
Oracle user: oracle
Spfile: +DATA/CDB1911A/PARAMETERFILE/spfile.270.1111925731
Password file: +DATA/CDB1911A/PASSWORD/pwdcdb1911a.256.1111924723
...
Database instances: cdb1911a3,cdb1911a2,cdb1911a3
Configured nodes: oelc5n3,oelc5n2,oelc5n1
...
Database is administrator managed

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
6.0	6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0, 6.0.13.1, 6.0.14.0, 6.0.15.0
5.3	5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0, 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0
5.2	5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1
5.1	5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0
5.0	5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1, 5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

Resolution

Using the Delphix CLI align the Delphix Instance numbers with those held by Oracle.

Delphix currently holds instanceNumber:1 and Oracle has INST_NUMBER 3. 

dlpx6014 *> sourceconfig
dlpx6014 sourceconfig *> select cdb19c
dlpx6014 sourceconfig 'cdb19c' *> update
dlpx6014 sourceconfig 'cdb19c' update *> set instances.0.instanceNumber=3
dlpx6014 sourceconfig 'cdb19c' update *> commit

Attempt another snapsync of the dSource.

Troubleshooting

A mismatch between the instance numbers held by Oracle for each RAC instance and those held by the Delphix metadata exists.

For example, the cluster may start out with the following configuration:

SQL> col inst_name format a40
SQL> select inst_number,inst_name from v$active_instances;

INST_NUMBER INST_NAME
----------- ----------------------------------------
          1 oelc5n2.plb.internal:cdb1911a2
          2 oelc5n3.plb.internal:cdb1911a1
          3 oelc5n1.plb.internal:cdb1911a3

Instances 1 and 2 are removed from the RAC database.  This may have been performed through DBCA, srvctl or some other method and the Oracle configuration ends up as the following.

SQL> col inst_name format a40
SQL> select inst_number,inst_name from v$active_instances;

INST_NUMBER INST_NAME
----------- ----------------------------------------
          3 oelc5n1.plb.internal:cdb1911a3

The Delphix configuration is left with an instance or instances where the instance number(s) held by Delphix do not align with Oracle.

In this scenario Delphix holds instanceNumber:1 and Oracle has INST_NUMBER 3.

dlpx6015 sourceconfig 'cdb1911a'> ls
Properties
    type: OracleRACConfig
    name: cdb1911a
    cdbType: ROOT_CDB
    credentials: (unset)
    crsDatabaseName: cdb1911a
    databaseName: cdb1911a
    discovered: true
    environmentUser: delphix
    instances:
        0:
            type: OracleRACInstance
            instanceName: cdb1911a3
            instanceNumber: 1
            node: oelc5n1
...