How to Generate a Non-Maskable Interrupt (NMI) in Oracle Cloud Infrastructure (OCI) (KBA9343)

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
6.0	6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0, 6.0.13.1, 6.0.14.0, 6.0.15.0, 6.0.16.0

Issue 

There are two typical sets of symptoms when an Engine becomes unresponsive:

1. The Delphix Engine NFS mounts are not responsive, and all target host IO operations fail. The web interface will not load, and attempts to login via SSH are unsuccessful as any Delphix Admin or self-service user, and no login prompt is presented when SSH connection is attempted.
2. The Delphix Engine NFS mounts are responsive, and VDB operations are not disrupted.  The web interface will not load, and attempts to login via SSH are unsuccessful as any Delphix Admin or self-service user.  A login prompt is received when SSH connection is attempted, but login attempts fail, with no password prompt following the entry of a username.

In both conditions, the hypervisor still indicates the virtual machine (VM) is running, and ping may return successfully. Memory and CPU utilization may be variable, or the VM may indicate no activity.

In condition 1, a non-maskable interrupt (NMI) may be sent from the hypervisor to cause the Delphix operating system (DxOS) to kernel panic and generate a crash dump. The resulting crash dump can be collected by Delphix Support for further analysis.

If there is no response to the NMI on the VM console, retry the procedure. The final recourse is to reset or power on/off the system which will not generate a core and reduces potential for root cause analysis.

It is important to note that this procedure may not be successful in all cases. Unresponsive VM situations may occur for a variety of reasons related to the guest operating system or hypervisor issues. The following procedure is a best-effort to collect system state information at the time of a VM becoming unresponsive.

In condition 2, a Delphix Support user may still be able to login and offer recovery options. If possible, the NMI should not be issued or Engine rebooted until Support is engaged for further direction.

NMI / Diagnostic Interrupt Procedure

In the OCI Console, navigate to Compute > Instances, and locate the Delphix VM of concern.

1. Click on the instance name to view details.
   
    
2. Under More Actions drop-down menu, click Send Diagnostic Interrupt.
   
    
3. A confirmation dialog will be presented confirming the desired action, with disclaimers.
   
   
   Note: Although OCI offers Console history to display "OS-level error messages", there will be no output available here.
   
   The Metrics panel may indicate zero CPU utilization when the VM is in an unresponsive state, and you may observe an increase in CPU utilization once the VM is back online following the panic and reboot.

Following this activity, once the Engine is online and accessible, a Support log bundle can be collected through the normal interface, but the dump files generated will need to be transferred by a Delphix Support engineer via screen-sharing session for further RCA.