Oracle vDB start fails with ORA-12547: TNS:lost contact (KBA10883)

Issue

Starting an Oracle vDB fails with the following error:

Enable dataset "VDB".
Error
Failed to perform verification for virtual database "VDB" on host "hostname.delphix.com" due to error 
"exception.oracle.accessor.database.failed.connection {Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException:
Cannot get Connection from Datasource: java.sql.SQLException: ORA-12547: TNS:lost contact
}"
Error Code
exception.oracle.vdb.database.verification.failed
Suggested Action
Review and resolve the error and run the operation again.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Date	Release
Feb 21, 2024	20.0.0.0
Jan 25, 2024	19.0.0.0
Dec 20, 2023 | Jan 10, 2024	18.0.0.0 | 18.0.0.1
Nov 21, 2023	17.0.0.0
Oct 18, 2023	16.0.0.0
Sep 21, 2023	15.0.0.0
Aug 24, 2023	14.0.0.0
Jul 24, 2023	13.0.0.0
Jun 21, 2023	12.0.0.0
May 25, 2023	11.0.0.0
Apr 13, 2023	10.0.0.0 | 10.0.0.1
Mar 13, 2023 | Mar 20, 2023	9.0.0.0 | 9.0.0.1
Feb 13, 2023	8.0.0.0
Jan 12, 2023	7.0.0.0
Releases Prior to 2023
Major Release	All Sub Releases
6.0	6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0, 6.0.13.1, 6.0.14.0, 6.0.15.0, 6.0.16.0, 6.0.17.0, 6.0.17.1, 6.0.17.2

Resolution

This fault can be due to the VDB not being shutdown cleanly. To force a cleanup, complete the following procedure:

In Delphix, ensure the vDB is disabled.

On the target server:

1. mount | grepVDB (ensure there are no mounted filesystems)
2. df -h (ensure there are no stale NFS mounts)
3. ps -ef | grep pmon | grep [VDB] (ensure no pmon is running for the database)
4. Execute the following to cleanup the stale instance:

export ORACLE_HOME=<path for the oracle home>
export ORACLE_SID=<VDB>
$ORACLE_HOME/bin/sqlplus /nolog
SQL> set _prelim on
SQL> conn / as sysdba
Prelim connection established
SQL> shutdown abort

Troubleshooting

Check the previous shutdown in the Oracle alert log or target connector logs for any shutdown issues:

1. Example from the previous shutdown attempt captured in the Delphixconnector logs:

ERROR at line 1:
ORA-01012: not logged on
Process ID: 0
Session ID: 0 Serial number: 0'
Database is in bad state.

2. When connecting to the instance you do not see the expected Connected to Idle Instance.
   
   Example:

sqlplus '/as sysdba'
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Feb 28 08:30:18 2024 Version 19.17.0.0.0
Copyright (c) 1982, 2022, Oracle. All rights reserved.
Connected.
SQL> exit

This occurs even when no Oracle pmon process is running on the target.