During the provisioning of an Oracle pluggable database (VPDB), the provision fails and reports the error:
Error Failed to merge keystore at "/mnt/tde/tdekeystores/ractde1" and "/mnt/tde/tdekeystores/ractde2/tde" into the new keystore at "/mnt/tde/tdekeystores/oracle_tde_keystores/auxiliary_cdb_keystores/Cvrpdb17twJl/tde".
- The Oracle dSource container database has Transparent Data Encryption enabled and the keystore used by this has been placed in an Oracle ASM diskgroup.
- Provisioning of an Oracle virtual pluggable database is being attempted.
- The keystore from the dSource has been copied out of the ASM diskgroup using the ASM command line utility
asmcmd cpto a regular file system before being transferred to the VPDB target host to be used during provisioning.
- Delphix Virtualization Engine 184.108.40.206 (or later) is in use.
Applicable Delphix Versions
- Click here to view the versions of the Delphix engine to which this article applies
Major Release All Sub Releases 6.0
220.127.116.11, 18.104.22.168, 22.214.171.124
Do not use Oracle ASM
asmcmd to copy the dSource wallet/keystore out of the ASM diskgroup that the source database is using to hold its keystore.
To establish the source database keystore in the target host, ensure that the keystore/wallet contents are extracted from the source database keystore using sqlplus
administer key management commands.
In the source database itself perform the following:
- Create an OS file system based keystore that can then be transferred to the VPDB target host.
administer key management create keystore '/home/oracle/tdekeystore/' identified by delphix123;
- Export the keys from the ASM backed keystore and import them into the new file system based keystore created in the previous step.
administer key management merge keystore '+OCR/WALLET/RACTDE1/tde/' identified by delphix123 into existing keystore '/home/oracle/tdekeystore/' identified by delphix123 with backup;
- Confirm the OS based keystore can be read and contains the source database keys.
orapki wallet display -wallet /home/oracle/tdekeystore/ Oracle PKI Tool Release 126.96.36.199.0 - Production Version 188.8.131.52.0 Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved. Enter wallet password: Requested Certificates: Subject: CN=oracle User Certificates: Oracle Secret Store entries: ORACLE.SECURITY.ID.ENCRYPTION. ORACLE.SECURITY.KB.ENCRYPTION. Trusted Certificates:
Further detail regarding the correct approach to exporting the TDE keys from a keystore held in ASM diskgroups use the Delphix knowledge base article : Exporting a Keystore from ASM to a Target Host for Oracle TDE Provisioning ( KBA8286 )
Transfer the keystore files from new file system based keystore location (in this case "/home/oracle/tdekeystore/") to the VPDB target host placing it in the location that will be set as the Parent Keystore Location during the provision.
Delphix displays the following error at the time the provision fails:
Error Failed to merge keystore at "/mnt/tde/tdekeystores/ractde1" and "/mnt/tde/tdekeystores/ractde2/tde" into the new keystore at "/mnt/tde/tdekeystores/oracle_tde_keystores/auxiliary_cdb_keystores/Cvrpdb17twJl/tde". Error Code exception.oracle.tde.merge.into.new.keystore.failed Suggested Action Make sure that the parent TDE keystore password and/or target TDE keystore password are correct. Review error output for more details and retry the operation. Command Output StatementCallback; uncategorized SQLException for SQL [ADMINISTER KEY MANAGEMENT MERGE KEYSTORE '/mnt/tde/tdekeystores/ractde1' IDENTIFIED BY **** AND KEYSTORE '/mnt/tde/tdekeystores/ractde2/tde' IDENTIFIED BY **** INTO NEW KEYSTORE '/mnt/tde/tdekeystores/oracle_tde_keystores/auxiliary_cdb_keystores/Cvrpdb17twJl/tde' IDENTIFIED BY ****]; SQL state ; error code ; ORA-46637: cannot add first keystore to the target keystore ; nested exception is java.sql.SQLException: ORA-46637: cannot add first keystore to the target keystore
The error itself is indicating that there has been a problem merging the source database keystore and the destination CDB keystore to create a third keystore that will be used by the auxiliary database Delphix creates during the provisioning process.
Examining the trace file from the Oracle session in the auxiliary CDB where the failed administer key management command has been executed will show the following error:
Trace file /u01/app/oracle/diag/rdbms/cvrpdb1rcx89/Cvrpdb1rCx81/trace/Cvrpdb1rCx81_ora_26463.trc Oracle Database 19c Enterprise Edition Release 184.108.40.206.0 - Production Version 220.127.116.11.0 Build label: RDBMS_18.104.22.168.0DBRUR_LINUX.X64_210712 ORACLE_HOME: /u01/app/oracle/19.3 System name: Linux Node name: oelc5n1.plb.internal Release: 5.4.17-2011.0.7.el7uek.x86_64 Version: #2 SMP Mon Mar 16 20:48:30 PDT 2020 Machine: x86_64 Instance name: Cvrpdb1rCx81 Redo thread mounted by this instance: 2 Oracle process number: 9 Unix process pid: 26463, image: email@example.com *** 2022-06-23T15:04:24.177396+10:00 (CDB$ROOT(1)) *** SESSION ID:(128.17584) 2022-06-23T15:04:24.177417+10:00 *** CLIENT ID:() 2022-06-23T15:04:24.177420+10:00 *** SERVICE NAME:() 2022-06-23T15:04:24.177422+10:00 *** MODULE NAME:(firstname.lastname@example.org (TNS V1-V3)) 2022-06-23T15:04:24.177424+10:00 *** ACTION NAME:() 2022-06-23T15:04:24.177427+10:00 *** CLIENT DRIVER:(jdbcoci : 22.214.171.124.0) 2022-06-23T15:04:24.177429+10:00 *** CONTAINER ID:(1) 2022-06-23T15:04:24.177432+10:00 kztsmcombine: could not load the source with error 29106
Oracle indicates the following in regards to this error code:
$ oerr ora 29106 29106, 00000, "Cannot import PKCS #12 wallet." // *Cause: A required parameter is NULL or the BER-encoding is malformed. // *Action: Enable tracing and attempt the connection again. Contact // Oracle customer support with the trace output.
Determining where the failure actually lies requires reading each of the keystores involved in the merge process using the Oracle supplied tool
$ orapki wallet display -wallet /mnt/tde/tdekeystores/ractde1/ewallet.p12 -pwd delphix123 -complete Oracle PKI Tool Release 126.96.36.199.0 - Production Version 188.8.131.52.0 Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved. Got tag 3 instead of 16.
The Got tag 3 instead of 16 is indicating that the wallet file is malformed and has been copied out of an ASM diskgroup using
asmcmd cp instead of sqlplus based
Further detail regarding this can be found in the following Oracle MOS notes:
- Cannot open wallet from local filesystem after cp from ASM with asmcmd (Doc ID 2085607.1)
- How to Manage a TDE wallet created in ASM on primary and to copy on Standby in ASM (Doc ID 2251874.1)
The following articles may provide more information or related information to this article:
- Exporting a Keystore from ASM to a Target Host for Oracle TDE Provisioning ( KBA8286 )
- Oracle MOS note : Cannot open wallet from local filesystem after cp from ASM with asmcmd (Doc ID 2085607.1)
- Oracle MOS note : How to Manage a TDE wallet created in ASM on primary and to copy on Standby in ASM (Doc ID 2251874.1)