SnapSync of Oracle Database Fails Due to Oracle Error ORA-27211 (KBA1773)
Applicable Delphix Versions
All Versions
Issue
When running a SnapSync job against an Oracle database, it may fail with an error that indicates RMAN cannot load the SBT shared library residing under the Delphix toolkit directory:
event_description | DB_SYNC job for "RM_Dsrc/OMVFPDP1" failed due to an error during execution: RMAN cannot load the SBT Library for database "OMVFPDP1". event_response | event_action | This error usually indicates a lack of read or execute permissions for user "oracle" in a directory in the path to the MML library located at "/apps/opt/delphix_os/toolkit/Delphix_42073c46_c94d_e5db_6d04_2f83ec5a7c53_delphix_os_cluster/toolkits/link/oracle/linux/x86/lib64" in host "144.7.7.162".
The detailed output of the error message shows the specific RMAN command, path to the library, library name and error message:
| RMAN> RUN {
| 2> # add block options
| 3> SET NOCFAU;
| 4> SQL 'ALTER SYSTEM CHECKPOINT GLOBAL';
| 5> SQL 'ALTER SYSTEM ARCHIVE LOG CURRENT';
| 6>
| 7> ALLOCATE CHANNEL DLPX_CHANNEL_0 DEVICE TYPE sbt
| 8> PARMS='SBT_LIBRARY=/apps/opt/delphix_os/toolkit/Delphix_42073c46_c94d_e5db_6d04_2f83ec5a7c53_delphix_os_cluster/toolkits/link/oracle/linux/x86/lib64/libobk_proxy.so,
| 9> BLKSIZE=262144,
| 10> ENV=(
| 11> DELPHIX_CONFIG_DIR=/apps/opt/delphix_os/toolkit/Delphix_42073c46_c94d_e5db_6d04_2f83ec5a7c53_delphix_os_cluster/toolkits/link/oracle/linux/x86/lib64,
| 12> DELPHIX_SERVER=delphix_server_name,
| 13> DELPHIX_INSTANCE=SNL.1.55.POGLXXVgiqq5KlDg2qPhZ8B,
| 14> DELPHIX_CHANNEL=DLPX_CHANNEL_0,
| 15> DELPHIX_PROXY=127.0.0.1,
| 16> DELPHIX_PORT=27434
| 17> )';
...etc...
| executing command: SET NOCFAU
|
| sql statement: ALTER SYSTEM CHECKPOINT GLOBAL
|
| sql statement: ALTER SYSTEM ARCHIVE LOG CURRENT
|
| RMAN-00571: ===========================================================
| RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
| RMAN-00571: ===========================================================
| RMAN-03009: failure of allocate command on DLPX_CHANNEL_0 channel at 08/27/2018 09:48:03
| ORA-19554: error allocating device, device type: SBT_TAPE, device name:
| ORA-27211: Failed to load Media Management Library
| Additional information: 3970
|
|
| RMAN>
|
| RMAN> exit
Delphix uses the System Backup to Tape (SBT) library to perform a nondisk backup directly to the Delphix engine. If this library can't be loaded, the RMAN command fails.
Troubleshooting
When performing a SnapSync against a RAC environment, Delphix randomly runs the SnapSync job against one of the RAC nodes. It is important to perform the following test on all nodes.
If the dSource is a single instance, the test can be performed on the one server hosting the source database.
As the Oracle instance owner (see the user name referenced in the error message), try running the "ldd" command to make sure the user has read and execute permission. If the user does not have the proper permissions, it will fail:
$ ldd /apps/opt/delphix_os/toolkit/Delphix_42073c46_c94d_e5db_6d04_2f83ec5a7c53_delphix_os_cluster/toolkits/link/oracle/linux/x86/lib64/libobk_proxy.so ldd: /apps/opt/delphix_os/toolkit/Delphix_42073c46_c94d_e5db_6d04_2f83ec5a7c53_delphix_os_cluster/toolkits/link/oracle/linux/x86/lib64/libobk_proxy.so: No such file or directory
Whereas, if the user has the correct permissions it will succeed:
$ ldd /apps/opt/delphix_os/toolkit/Delphix_42073c46_c94d_e5db_6d04_2f83ec5a7c53_delphix_os_cluster/toolkits/link/oracle/linux/x86/lib64/libobk_proxy.so
linux-vdso.so.1 => (0x00007ffc609aa000)
libc.so.6 => /lib64/libc.so.6 (0x00007f51e102a000)
/lib64/ld-linux-x86-64.so.2 (0x0000003df5c00000)
Resolution
Grant the user read and execute permission so that the "ldd" command succeeds:
chmod 750 /apps/opt/delphix_os chmod 770 /apps/opt/delphix_os/toolkit
$ ldd /apps/opt/delphix_os/toolkit/Delphix_42073c46_c94d_e5db_6d04_2f83ec5a7c53_delphix_os_cluster/toolkits/link/oracle/linux/x86/lib64/libobk_proxy.so
linux-vdso.so.1 => (0x00007ffc609aa000)
libc.so.6 => /lib64/libc.so.6 (0x00007f51e102a000)
/lib64/ld-linux-x86-64.so.2 (0x0000003df5c00000)
External Links