TB020 Non-RDBMS Activity May Result in Delphix Engine Restarts

Alert Type

Availability

Impact

The Delphix Engine may unexpectedly and intermittently reboot. Access to virtual databases (VDBs) may be temporarily suspended, and affected VDBs may hang or crash. Delphix jobs or actions running at the time of the failure, e.g. SnapSync or Provisions, may fail when interrupted by a reboot.

Contributing Factors

The issue may occur in the following Delphix Releases:

All Delphix Engine 3.0 releases
All Delphix Engine 3.1 releases
All Delphix Engine 3.2 releases
All Delphix Engine 4.0 releases
All Delphix Engine 4.1 releases
Delphix Engine 4.2.1.0 and 4.2.1.1

The problem can only occur on Delphix filesystems provisioned via NFS, such as those used by VDBs on UNIX/Linux hosts and unstructured files using vFiles.

There is a low incidence of occurrence of this issue. Only specific, rarely used I/O patterns can trigger the problem, such as truncating then subsequently re-writing an existing file. These are not patterns that occur with databases such as Oracle; however, they can occur when writing other types of non-database files or unstructured data on a Delphix VDB filesystem.

The problem is known to be triggered by storing Microsoft Excel spreadsheets on a Delphix filesystem from UNIX/Linux hosts.

iSCSI volumes, such as used by SQL Server VDBs and vFiles on Windows, are not susceptible.

Symptoms

Following an unexpected reboot, an alert will be created with the following descriptive text
```
Unexpected server restart. The server is starting up following an unexpected shutdown around <date>. Contact Delphix Support.
```
Note: this message can also occur if the Delphix Engine guest is restarted from the VMware vSphere™ Client.
Jobs running at the time of the failure may fail with the alert:
```
<job_type> for "<object>" failed due to server restart during execution
```
where <job_type> is the type of job running, foe example, DB_REFRESH, DB_PROVISION, or DB_SYNC and <object> is the name of the Delphix group and database name for which the job was being processed.
When a restart occurs VDBs may experience a temporary suspension of service. SQL Server VDBs may be inaccessible until they are restarted. In the system log of affected Oracle target hosts messages like:
```
NFS server <ip address> not responding
```
```
may be seen on the console or in the system log, where <ip address> is the network address of the affected Delphix Engine. 
```

Relief/Workaround

Avoid use of an Oracle VDB filesystem for storing unstructured data, i.e. non-RDBMS, files.

Avoid affected I/O patterns on NFS-backed VDB containers on Delphix, including those used for Oracle VDBs or vFiles. This includes the use of applications which routinely truncate and then re-write files, such as text editors and spreadsheet programs, when they are writing to an susceptible VDB container.

Resolution

The issue is fully resolved in Delphix Engine OS Version 4.2.2015.04.04, included with Delphix Engine 4.2.2.0.

Fresh Installations and Full Upgrades to Delphix Engine 4.2.2.0 will run OS Version 4.2.2015.04.04.

Deferred upgrades to Delphix 4.2.2.0 run a prior version of the Delphix OS and are still susceptible to the problem described in this bulletin.

See the "Deferred OS Upgrade" section of "Upgrading to a New Version of the Delphix Engine" for information about how to determine the currently running OS version of a Delphix Engine.