How to Generate a Non-Maskable Interrupt (NMI) in Microsoft Azure (KBA4327)

Issue

There are two typical sets of symptoms when an Engine becomes unresponsive:

1. The Delphix Engine NFS mounts are not responsive, and all target host IO operations fail. The web interface will not load, and attempts to login via SSH are unsuccessful as any Delphix Admin or self-service user, and no login prompt is presented when SSH connection is attempted.

2. The Delphix Engine NFS mounts are responsive, and VDB operations are not disrupted.  The web interface will not load, and attempts to login via SSH are unsuccessful as any Delphix Admin or self-service user.  A login prompt is received when SSH connection is attempted, but login attempts fail, with no password prompt following the entry of a username.

In both conditions, the hypervisor still indicates the virtual machine (VM) is running, and ping may return successfully. Memory and CPU utilization may be variable, or the VM may indicate no activity.

In condition 1, a non-maskable interrupt (NMI) may be sent from the hypervisor to cause the Delphix operating system (DxOS) to kernel panic and generate a crash dump. The resulting crash dump can be collected by Delphix Support for further analysis.

If there is no response to the NMI on the VM's console, retry the procedure. The final recourse is to reset or power on/off the system which will not generate a core and reduces potential for root cause analysis.

It is important to note that this procedure may not be successful in all cases. Unresponsive VM situations may occur for a variety of reasons related to the guest operating system or hypervisor issues. The following procedure is a best-effort to collect system state information at the time of a VM becoming unresponsive.

In condition 2, a Delphix Support user may still be able to login and offer recovery options so if possible, the NMI should not be issued or Engine rebooted until Support is engaged for further direction.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
6.0	6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0, 6.0.13.1, 6.0.14.0
5.3	5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0, 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0
5.2	5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1
5.1	5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0
5.0	5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1, 5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

NMI Procedure

1. After logging into the Azure Portal, navigate to "Virtual Machines", then click the VM name in the resulting tab.

2. Scroll to the bottom of VM tools to locate "Support + troubleshooting" heading. Click "Boot diagnostics" and click "Serial log" to access the VM serial log history.  A "Download serial log" hyperlink should be available to download the current console content prior to the NMI operation. This serial log access is also helpful for monitoring the NMI and resulting panic, and any startup messages post-restart.

Missing permissions for storage blob container may result in an error:

Error encountered while getting the screenshot or serial log file from the blob container in storage account <storage account name>. Please make sure you have permissions and fireall (sic) is not blocking access to the Storage account.

3. From "Support + troubleshooting", click "Serial console" to access the VM interactive console.  Initially there will be a delay for a number of seconds while the console connects:

Missing permissions for the admin user will cause the Serial console connection may result in an error:

The serial console connection to the VM encountered an error: 'Forbidden (403) - You do not have the required permissions to use this VM serial console. Please ensure you have at least VM Contributor role permissions.

4. Once connected, the "Send Command" button will be available. Click this icon, then click "Send Non-Maskable Interrupt (NMI)".

A final warning will be posted to the user that the VM will be crashed and restarted for debugging purposes. Click "Send NMI" button to initiate the process.

Monitoring Delphix VM Console during NMI

The serial console should subsequently indicate a VM panic due to NMI received.

panic[cpu0]/thread=ffffff003d005c40: NMI received

fffffffffbc18ed0 fffffffffbad559f ()
fffffffffbc18f00 unix:av_dispatch_nmivect+34 ()
fffffffffbc18f10 unix:nmiint+152 ()
ffffff003d005bd0 unix:mach_cpu_idle+6 ()
ffffff003d005c00 unix:cpu_idle+11a ()
ffffff003d005c20 unix:idle+a7 ()
ffffff003d005c30 unix:thread_start+8 ()

dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel

[   16.646585] kdump-tools[702]: Starting kdump-tools:
[   16.656098] kdump-tools[716]:  * running makedumpfile -c -d 31 --message-level 22 --private-page-filter 0x2F5ABDF11ECAC4E /proc/vmcore /var/crash/202204281528/dump-incomplete
[   32.066802] kdump-tools[733]: Page filter: 0x2f5abdf11ecac4e
[   32.075554] kdump-tools[733]: STEP [Checking for memory holes  ] : 0.023162 seconds
[   32.083134] kdump-tools[733]: STEP [Excluding unnecessary pages] : 0.057384 seconds
[   32.089872] kdump-tools[733]: STEP [Copying data               ] : 14.745280 seconds
[   32.097036] kdump-tools[733]: STEP [Copying data               ] : 0.003801 seconds
[   32.104759] kdump-tools[733]: Original pages  : 0x00000000001e23bb
[   32.111072] kdump-tools[733]:   Excluded pages   : 0x00000000001af8dd
[   32.117443] kdump-tools[733]:     Pages filled with zero  : 0x0000000000032667
[   32.124092] kdump-tools[733]:     Non-private cache pages : 0x000000000003acf5
[   32.131174] kdump-tools[733]:     Private cache pages     : 0x00000000000000aa
[   32.137880] kdump-tools[733]:     private filter pages : 0x00000000000565d7
[   32.144447] kdump-tools[733]:     User process data pages : 0x000000000007b5b8
[   32.152035] kdump-tools[733]:     Free pages              : 0x0000000000070948
[   32.159405] kdump-tools[733]:     Hwpoison pages          : 0x0000000000000000
[   32.166963] kdump-tools[733]:     Offline pages           : 0x0000000000000000
[   32.174569] kdump-tools[733]:   Remaining pages  : 0x0000000000032ade
[   32.181013] kdump-tools[733]:   (The number of pages is reduced to 10%.)
[   32.188092] kdump-tools[733]: Memory Hole     : 0x00000000000ddc45
[   32.194236] kdump-tools[733]: --------------------------------------------------
[   32.200962] kdump-tools[733]: Total pages     : 0x00000000002c0000
[   32.208083] kdump-tools[733]: Cache hit: 514825, miss: 904, hit rate: 99.8%
[   32.215449] kdump-tools[733]: The dumpfile is saved to /var/crash/202204281528/dump-incomplete.
[   32.226237] kdump-tools[733]: makedumpfile Completed.
[   33.430492] kdump-tools[716]:  * kdump-tools: saved vmcore in /var/crash/202204281528
[   38.811794] kdump-tools[716]:  * running makedumpfile --dump-dmesg /proc/vmcore /var/crash/202204281528/dmesg.202204281528
[   38.831262] kdump-tools[737]: The dmesg log is saved to /var/crash/202204281528/dmesg.202204281528.
[   38.841395] kdump-tools[737]: makedumpfile Completed.
[   38.847671] kdump-tools[716]:  * kdump-tools: saved dmesg content in /var/crash/202204281528

Next Steps

Once this process is completed, and the VM is accessible again, Delphix Support will login to the Engine directly to relocate the crash dump file to be collected in a Support log bundle, or manually transferred from the VM via SCP, etc.