Resolving high CPU on Windows VDB hosts after applying Windows Updates (KBA6024)

Issue

After applying Windows updates or .NET updates for Windows Server 2016 or 2019, servers used to host Delphix VDBs may experience the following symptoms:

• CPU usage on the Windows target host may increase significantly, possibly reaching 100%
• powershell may take up to 10 seconds to launch from the command line

This issue has been observed after applying the following updates:

• Windows Server 2016, April 2020 Update (KB4550947) and later
• Windows Server 2019, April 2020 Update (KB4550969) and later

Some Windows updates may include bug and security fixes for Microsoft.NET. Applying these updates will cause optimized (compiled) versions of some .NET assemblies to become invalid once the server is restarted.

In order to monitor the health of VDBs, the Delphix Engine will invoke various scripts using PowerShell on a regular basis, once for each VDB, every 5 minutes.

Because PowerShell uses Microsoft.NET assemblies, the scripts issued as part of VDB monitoring may need to dynamically recompile these assemblies each time PowerShell is run. This can result in monitoring scripts that take up to 20 times longer than normal.

Depending on the number of VDBs, and the number of CPUs available on the Windows host, this can result in Delphix-initiated PowerShell  commands consuming up to 100% of available CPU on a Windows Staging or Target host for extended periods of time.

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
6.0	6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1
5.3	5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0, 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0
5.2	5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1
5.1	5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0
5.0	5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1, 5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

Resolution

In many cases, the .NET optimization service will trigger automatically, and the issue will self-resolve within a few hours of the Windows update being applied.

Occasionally, high CPU utilization or other system configuration prevents the .NET optimization service from running, including CPU usage from affected PowerShell processes. To resolve this, invalid assemblies must be recompiled manually.

The following commands can be run on affected servers to temporarily stop the Delphix Connector and recompile (optimize) any invalid .NET assemblies.

Open a DOS command prompt and execute the following commands from an account with "Administrator" privileges:

net stop "Delphix Connector"
C:
cd \Windows\Microsoft.NET\Framework64\v4.0.30319
.\ngen update
net start "Delphix Connector"

Note:

While the Delphix Connector is stopped, the Delphix Engine will be unable to monitor the status of related dSources and VDBs. Jobs on VDBs and Staging databases will fail until the Delphix Connector service is restarted.

To prevent future recurrences of these issue, consider running ngen update manually after applying Windows Updates.