TDE Enabled Virtual Pluggable Database Provisioning Fails Indicating a Failure to Merge TDE Keystores (KBA9304)

Issue

During the provisioning of an Oracle pluggable database (VPDB), the provision fails and reports the error:

Error
Failed to merge keystore at "/mnt/tde/tdekeystores/ractde1" and "/mnt/tde/tdekeystores/ractde2/tde" into the new keystore at "/mnt/tde/tdekeystores/oracle_tde_keystores/auxiliary_cdb_keystores/Cvrpdb17twJl/tde".

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
6.0	6.0.13.0, 6.0.13.1, 6.0.14.0

Resolution

Do not use Oracle ASM asmcmd to copy the dSource wallet/keystore out of the ASM diskgroup that the source database is using to hold its keystore. 

To establish the source database keystore in the target host, ensure that the keystore/wallet contents are extracted from the source database keystore using sqlplus administer key management commands.

In the source database itself perform the following:

1. Create an OS file system based keystore that can then be transferred to the VPDB target host.

administer key management create keystore '/home/oracle/tdekeystore/' identified by delphix123;

2. Export the keys from the ASM backed keystore and import them into the new file system based keystore created in the previous step.

administer key management merge keystore '+OCR/WALLET/RACTDE1/tde/' identified by delphix123 into existing keystore '/home/oracle/tdekeystore/' identified by delphix123 with backup;

3. Confirm the OS based keystore can be read and contains the source database keys.

orapki wallet display -wallet /home/oracle/tdekeystore/

Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:   
Requested Certificates: 
Subject:        CN=oracle
User Certificates:
Oracle Secret Store entries: 
ORACLE.SECURITY.ID.ENCRYPTION.
ORACLE.SECURITY.KB.ENCRYPTION.
Trusted Certificates: 

Further detail regarding the correct approach to exporting the TDE keys from a keystore held in ASM diskgroups use the Delphix knowledge base article : Exporting a Keystore from ASM to a Target Host for Oracle TDE Provisioning ( KBA8286 )

Transfer the keystore files from new file system based keystore location (in this case "/home/oracle/tdekeystore/") to the VPDB target host placing it in the location that will be set as the Parent Keystore Location during the provision.

Troubleshooting

Delphix displays the following error at the time the provision fails:

Error
Failed to merge keystore at "/mnt/tde/tdekeystores/ractde1" and "/mnt/tde/tdekeystores/ractde2/tde" into the new keystore at "/mnt/tde/tdekeystores/oracle_tde_keystores/auxiliary_cdb_keystores/Cvrpdb17twJl/tde".

Error Code
exception.oracle.tde.merge.into.new.keystore.failed

Suggested Action
Make sure that the parent TDE keystore password and/or target TDE keystore password are correct. Review error output for more details and retry the operation.

Command Output
 StatementCallback; uncategorized SQLException for SQL [ADMINISTER KEY MANAGEMENT MERGE KEYSTORE '/mnt/tde/tdekeystores/ractde1' IDENTIFIED BY **** AND KEYSTORE '/mnt/tde/tdekeystores/ractde2/tde' IDENTIFIED BY **** INTO NEW KEYSTORE '/mnt/tde/tdekeystores/oracle_tde_keystores/auxiliary_cdb_keystores/Cvrpdb17twJl/tde' IDENTIFIED BY ****]; SQL state [99999]; error code [46637]; ORA-46637: cannot add first keystore to the target keystore
; nested exception is java.sql.SQLException: ORA-46637: cannot add first keystore to the target keystore

The error itself is indicating that there has been a problem merging the source database keystore and the destination CDB keystore to create a third keystore that will be used by the auxiliary database Delphix creates during the provisioning process.

Examining the trace file from the Oracle session in the auxiliary CDB where the failed administer key management command has been executed will show the following error:

Trace file /u01/app/oracle/diag/rdbms/cvrpdb1rcx89/Cvrpdb1rCx81/trace/Cvrpdb1rCx81_ora_26463.trc
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.11.1.0.0
Build label:    RDBMS_19.11.0.0.0DBRUR_LINUX.X64_210712
ORACLE_HOME:    /u01/app/oracle/19.3
System name:    Linux
Node name:      oelc5n1.plb.internal
Release:        5.4.17-2011.0.7.el7uek.x86_64
Version:        #2 SMP Mon Mar 16 20:48:30 PDT 2020
Machine:        x86_64
Instance name: Cvrpdb1rCx81
Redo thread mounted by this instance: 2
Oracle process number: 9
Unix process pid: 26463, image: oracle@oelc5n1.plb.internal

*** 2022-06-23T15:04:24.177396+10:00 (CDB$ROOT(1))
*** SESSION ID:(128.17584) 2022-06-23T15:04:24.177417+10:00
*** CLIENT ID:() 2022-06-23T15:04:24.177420+10:00
*** SERVICE NAME:() 2022-06-23T15:04:24.177422+10:00
*** MODULE NAME:(java@oelc5n1.plb.internal (TNS V1-V3)) 2022-06-23T15:04:24.177424+10:00
*** ACTION NAME:() 2022-06-23T15:04:24.177427+10:00
*** CLIENT DRIVER:(jdbcoci : 19.12.0.0.0) 2022-06-23T15:04:24.177429+10:00
*** CONTAINER ID:(1) 2022-06-23T15:04:24.177432+10:00

kztsmcombine: could not load the source with error 29106

Oracle indicates the following in regards to this error code:

$ oerr ora 29106
29106, 00000, "Cannot import PKCS #12 wallet."
// *Cause:  A required parameter is NULL or the BER-encoding is malformed.
// *Action: Enable tracing and attempt the connection again. Contact
//          Oracle customer support with the trace output.

Determining where the failure actually lies requires reading each of the keystores involved in the merge process using the Oracle supplied tool orapki.  

$ orapki wallet display -wallet /mnt/tde/tdekeystores/ractde1/ewallet.p12  -pwd delphix123 -complete
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.


Got tag 3 instead of 16.

The Got tag 3 instead of 16 is indicating that the wallet file is malformed and has been copied out of an ASM diskgroup using asmcmd cp instead of sqlplus based administer key managementcommands.

Further detail regarding this can be found in the following Oracle MOS note:

• How to Manage a TDE wallet created in ASM on primary and to copy on Standby in ASM (Doc ID 2251874.1)

 

 

 

---