Oracle 19c Cluster Fails to Discover Scan Listener (KBA5579)

Issue

Cluster discovery fails to discover Oracle Scan listener endpoints. 

This can result in issues with listener registration and JDBC connectivity issues when the SCAN listener has no default 1521 endpoint.

Error
Cannot connect to database due to: Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException: Cannot get Connection from Datasource: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
Error Code
exception.oracle.dbc.unmanaged.database.failed.connection
Suggested Action
Make sure that the username provided has access rights to system views, and that the host server can be reached.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
6.0	6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1
5.3	5.3.5.0, 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0

Resolution

A known bug results in SCAN endpoint discovery failing in Oracle cCusterware 19c and above. This is due to a change in the output format of the "srvctl config scan_listener" command.

If no scan listener endpoints are discovered during cluster discovery, the scan address with port 1521 will be used. If the scan listener is using this address then vdb snapshots should be successful, however if the scan listener is using a non-default port, the listener registration will fail and the engine will be unable to connect to the VDB via JDBC.

To work around this issue, the cluster can have a manual scan listener added using the CLI:

engine_name> /environment/oracle/listener
engine_name environment oracle listener> create
engine_name environment oracle listener create *> set type=OracleScanListener
engine_name environment oracle listener create *> set name=<listener name>
engine_name environment oracle listener create *> set environment=<cluster environment>
engine_name environment oracle listener create *> set protocolAddresses="(ADDRESS=(PROTOCOL=tcp)(HOST=<hostname or IP>)(PORT=<port#>))"
engine_name environment oracle listener create *> commit

Troubleshooting

After provisioning, the VDB should be available, however snapshots will fail. After connecting to the VDB via sqlplus, you can verify which remote listener the engine has used

SQL>show parameter remote_listener

NAME                     TYPE     VALUE
------------------------------------ ----------- ------------------------------
remote_listener              string     test1-scan.delphix.com:1521

If this does not match the endpoints returned from srvctl, then it is likely the environment is experiencing this issue.

$ORACLE_HOME/bin/srvctl config scan_listener
SCAN Listeners for network 1:
Registration invited nodes:
Registration invited subnets:
Endpoints: TCP:1531
SCAN Listener LISTENER_SCAN1 exists
SCAN Listener is enabled.

Note:

In the above example, the SCAN listener is using port 1531; however, due to the endpoint discovery failure, the engine is using the default 1521 port.