Oracle TDE provision fails with ORA-46636: cannot add second keystore to the target keystore (KBA9871)

Issue

Attempting to provision a TDE enabled virtual pluggable database (vPDB)  into a linked container database (CDB) fails with the following error:

ORA-46636: cannot add second keystore to the target keystore

An example of the entire error shown in Delphix for the provision/refresh job is provided below:

Error
Failed to merge keystore at "/mnt/provision/nfs_tde_source_keys" and "/u01/app/oracle/admin/CDOMLOSR60041/encryption_keystore" into the new keystore at "/u01/app/oracle/admin/CDOMLOSR60041/oracle_tde_keystores/auxiliary_cdb_keystores/CsdPDB3EWlrX/tde".

Error Code
exception.oracle.tde.merge.into.new.keystore.failed

Suggested Action
Make sure that the parent TDE keystore password and/or target TDE keystore password are correct. Review error output for more details and retry the operation.

Command Output
StatementCallback; uncategorized SQLException for SQL [ADMINISTER KEY MANAGEMENT MERGE KEYSTORE '/mnt/provision/nfs_tde/source_keys' IDENTIFIED BY **** AND KEYSTORE '/u01/app/oracle/admin/CDOMLOSR60041/encryption_keystore' IDENTIFIED BY **** INTO NEW KEYSTORE '/u01/app/oracle/admin/CDOMLOSR60041/oracle_tde_keystores/auxiliary_cdb_keystores/CsdPDB3EWlrX/tde' IDENTIFIED BY ****]; SQL state [99999]; error code [46636]; ORA-46636: cannot add second keystore to the target keystore ; nested exception is java.sql.SQLException: ORA-46636: cannot add second keystore to the target keystore

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
7.0	7.0.0
6.0	6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0, 6.0.13.1, 6.0.14.0, 6.0.15.0, 6.0.16.0, 6.0.17.0, 6.0.17.1, 6.0.17.2

Resolution

The error indicates that the problem is with the second keystore. In the example provided, the problematic keystore is the following:

/u01/app/oracle/admin/CDOMLOSR60041/encryption_keystore

This is the linked CDB keystore, in this case with the ORACLE_SIDCDOMLOSR60041.

Note:

It is recommended to use mkstore versus okapki, as with okapki, if there is an .sso file in the directory, then an invalid password will not be caught because the sso file will be used.

 

[dlpxqa@ip-10-110-228-84 tde]$ mkstore -wrl . -list
Oracle Secret Store Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:
./ewallet.p12 (Permission denied)

 

Note:

You should run mkstore as the user that owns the wallet, otherwise it will fail and leave a .lck file which will cause mkstore to fail when run as the correct owner of the wallet. This .lck file can be removed, Example error when previously run as dlpxqa user below.

[oracle@ip-10-110-228-84 tde]$ mkstore -wrl . -list
Oracle Secret Store Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:
Failed to lock...
attempt 1: java.io.FileNotFoundException: /work/tde/blackbox/CDOMSHTG663E/tde/ewallet.p12.lck (Permission denied)
attempt 2: java.io.FileNotFoundException: /tmp/pki_data-1188720789.lck (No such file or directory)
[oracle@ip-10-110-228-84 tde]$

In the above example, you can simply remove the .lck file owned by dlpxqa user:

[dlpxqa@ip-10-110-228-84 tde]$ ls -la
total 12
drwxrwxr-x. 2 oracle oinstall  103 Dec  6 22:10 .
drwxr-xr-x. 3 oracle oinstall   17 Dec  6 20:35 ..
-rw-------. 1 oracle oinstall 4040 Dec  6 20:35 cwallet.sso
-rw-------. 1 oracle oinstall 2555 Dec  6 20:35 ewallet_2023120701352441.p12
-rw-------. 1 oracle oinstall 3995 Dec  6 20:35 ewallet.p12
-rw-------. 1 dlpxqa oinstall    0 Dec  6 22:10 ewallet.p12.lck

1. Make sure the password provided for the linked CDB is correct in the Delphix UI. You can locate this by navigating to Environments > Databases in the GUI. 
   
   
   
   You can use Oracle tools such as mkstore to validate the password. For example, while logged in as the Delphix OS user, the following shows that the password is incorrect.

Note:

Starting in Oracle 18c, the TDE_ENCRYPTION can be used in leu of sqlnet.ora which avoids certain issues that can arise if TNS_ADMIN or sqlnet.ora is not setup correctly.

2. If you use sqlnet.ora , make sure the location of the wallet is correct and that TNS_ADMIN environment variable is setup correctly for the Delphix Environment user.

   ENCRYPTION_WALLET_LOCATION =
   (DIRECTORY=/u01/app/oracle/admin/CDOMLOSR60041/encryption_keystore))
   

    

 

See the Oracle documentation for using TDE_ENCRYPTION. 

See the following Oracle knowledge article about how to convert from using sqlnet.ora to wallet_root initialization parameter, "How To Convert From Using SQLNET.ENCRYPTION_WALLET_LOCATION To 19c Parameter ( WALLET_ROOT and TDE_CONFIGURATION) (Doc ID 2642694.1)".

 

---