VDB Refresh Performance With Direct NFS Degraded Following Oracle Upgrade (KBA8761)
KBA
KBA# 8761Applicable Delphix Versions
- Click here to view the versions of the Delphix engine to which this article applies
-
Major Release All Sub Releases 6.0 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0
Troubleshooting
An Oracle VDB using Direct NFS experiences performance degradation following an upgrade of the Oracle home in which it is running.
When starting a VDB or during refresh or provision jobs, if the target host meets the requirements for NFSv4 to be used and Oracle Direct NFS is enabled for the Oracle home, Delphix adds entries to the $ORACLE_HOME/dbs/oranfstab file defining the VDBs filesystem mounts. The Delphix OS user therefore needs write permission to the $ORACLE_HOME/dbs/oranfstab file on the target host.
If the write permission of the Delphix OS user is removed from the oranfstab file, it can no longer be updated when the filesystem mounts change during a VDB refresh job, leaving stale entries in place for the pre-refresh filesystems. This may occur if the ownership of the file is changed during Oracle home maintenance or upgrade work.
Having stale entries in the oranfstab can lead Direct NFS to repeatedly attempt to mount the old filesystems which may no longer be shared from the Delphix engine.
When this issue occurs, the VDB alert log shows repeated Direct NFS channel "UP" messages with no corresponding channel "DOWN" messages. Below is an example of this happening during a VDB refresh, as the media recovery begins. For long refresh jobs, these messages may repeat many times.
2022-01-12T05:20:29.247909-05:00 Serial Media Recovery started 2022-01-12T05:20:29.265376-05:00 Direct NFS: channel id [0] path [10.0.0.201] to filer [vdb10] via local [] is UP 2022-01-12T05:20:29.280245-05:00 Direct NFS: channel id [0] path [10.0.0.201] to filer [vdb10] via local [] is UP 2022-01-12T05:20:29.293713-05:00 Direct NFS: channel id [0] path [10.0.0.201] to filer [vdb10] via local [] is UP 2022-01-12T05:20:29.306790-05:00 Direct NFS: channel id [0] path [10.0.0.201] to filer [vdb10] via local [] is UP ORA-279 signalled during: alter database recover database noparallel until cancel using backup controlfile... alter database recover cancel 2022-01-12T05:20:29.367862-05:00 Direct NFS: channel id [0] path [10.0.0.201] to filer [vdb10] via local [] is UP 2022-01-12T05:20:29.379952-05:00 Direct NFS: channel id [0] path [10.0.0.201] to filer [vdb10] via local [] is UP 2022-01-12T05:20:29.392978-05:00 Direct NFS: channel id [0] path [10.0.0.201] to filer [vdb10] via local [] is UP 2022-01-12T05:20:29.404966-05:00 Direct NFS: channel id [0] path [10.0.0.201] to filer [vdb10] via local [] is UP
An Oracle 10046 trace of the VDB shows multiple occurrences of the following message.
kgnfs_mntrps:10078: KGNFS_NFSPROC3_MNT FAIL 13
Error code 13 translates to "permission denied". Details of how to perform a 10046 trace on a running session are available in the following Oracle document.
How To Collect 10046 Trace (SQL_TRACE) Diagnostics for Performance Issues (Doc ID 376442.1)
Confirming the Cause
The issue can be confirmed by comparing the mount entries for the affected VDB in the $ORACLE_HOME/dbs/oranfstab file with the filesystems that have been mounted by Delphix for a running refresh job.
In this example, the filesystem entries in the oranfstab file all start with '/domain0/group-3/oracle_db_container-7/oracle_timeflow-8'.
[oracle@gi-rhel77 dbs]# cat oranfstab ###CONFIGURATION ADDED BY DELPHIX 564d05d6_90e1_0db8_96de_2ffd9d93ead7 server: vdb10 path: 10.0.0.201 security_default: sys mnt_timeout: 90 nfs_version: nfsv4 export: /domain0/group-3/oracle_db_container-7/oracle_timeflow-8/datafile mount: /mnt/provision/vdb10/datafile export: /domain0/group-3/oracle_db_container-7/oracle_timeflow-8/archive mount: /mnt/provision/vdb10/archive export: /domain0/group-3/oracle_db_container-7/oracle_timeflow-8/temp mount: /mnt/provision/vdb10/temp export: /domain0/group-3/oracle_db_container-7/oracle_timeflow-8/source-archive mount: /mnt/provision/vdb10/source-archive
Checking the paths of the NFS filesystems mounted by Delphix for the running refresh job, we see that they have changed and now start with '/domain0/group-3/oracle_db_container-7/oracle_timeflow-260'.
[oracle@gi-rhel77 dbs]$ mount | grep vdb10 10.0.0.201:/domain0/group-3/oracle_db_container-7/oracle_timeflow-260 on /mnt/provision/vdb10 type nfs4 (rw,nosuid,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.43.76.27,local_lock=none,addr=10.0.0.201)
The entries in $ORACLE_HOME/dbs/oranfstab still refer to filesystems from a previous timeflow that are no longer being shared by the Delphix engine, hence the permission denied errors when Direct NFS attempts to mount them.
The repeated mount attempts add overhead which can impact the performance of the VDB that may manifest as longer than expected refresh jobs.
Prerequisites
- Oracle Direct NFS is enabled for the Oracle home in use by the VDB.
- The target host meets the requirements for NFSv4 to be used for the Delphix filesystem mounts.
- File ownership or permissions have changed in the affected Oracle home.
Resolution
To resolve this issue, confirm that the Delphix OS user on the affected target host has write permission on the $ORACLE_HOME/dbs/oranfstab file.
Once the Delphix OS user can again write to the $ORACLE_HOME/dbs/oranfstab file, any stale entries will be removed during the next start or refresh job for an affected VDB.
Related Articles
The following articles may provide more information or related information to this article: