TB102 Linux and UNIX Hosted Virtual Databases May Become Unresponsive and Crash

Impact

Rarely during certain virtual database (VDB) operations, all VDBs may become unresponsive and crash.  Affected operations include disable, refresh, and delete.

Contributing Factors

This article applies to the following versions of the Delphix Continuous Data engine (Virtualization):

Any VDB using the Network Filesystem (NFS) version 3 protocol may be impacted. This includes any VDBs hosted on Linux or UNIX target environments. 

VDBs exclusively using NFS version 4 protocol are not impacted. 

The issue does not impact Microsoft SQL Server or other environments on Windows.

In order for the problem to occur, a system must have been upgraded to one of the affected releases enumerated above. An initial installation on any of the above releases that has not been upgraded is not impacted. 

The issue is more likely to occur on Delphix 9.0.0.0 and may occur immediately following an upgrade. This is why the Delphix 9.0.0.0 image is no longer available for download.

Symptoms

Affected VDBs will hang or crash. 

Target environments may experience errors like “NFS Server <> not responding” displayed in the system log or console.

Relief/Workaround

In order to prevent this problem, use one of the following (in preference order):

1. Upgrade to Delphix release 9.0.0.1.

2. If you have already upgraded to an affected release, perform a “delay the reboot”-type (or deferred) upgrade to Delphix 9.0.0.1 to prevent the issue from occurring. 

3. Disable and re-enable all VDBs.

4. Set a global option to force NFS version 3 protocol (thereby eliminating the possibility of disable), as per the CLI example below:

demo-engine> service nfs
demo-engine service nfs> get
    type: NfsConfig
    mountVersion: AUTOMATIC
demo-engine service nfs> update
demo-engine service nfs update *> set mountVersion=NFSv3
demo-engine service nfs update *> commit
demo-engine service nfs> get
    type: NfsConfig
    mountVersion: NFSv3

Once the problem has occurred, the only remedy is to disable then re-enable all impacted VDBs.

Related Documents

None