Skip to main content
Delphix

Resolving high CPU on Windows VDB hosts after applying Windows Updates (KBA6024)

 

KBA

KBA# 6024

Issue

After applying Windows updates or .NET updates for Windows Server 2016 or 2019, servers used to host Delphix VDBs may experience the following symptoms:

  • CPU usage on the Windows target host may increase significantly, possibly reaching 100%
  • powershell may take up to 10 seconds to launch from the command line

This issue has been observed after applying the following updates:

  • Windows Server 2016 April (KB4550947), May (KB4556813) or June (KB4561616) Cumulative Updates, and later
  • Windows Server 2019 April (KB4550969), May (KB4551853) or June (KB4561608) Cumulative Updates, and later

Some Windows updates may include bug and security fixes for Microsoft.NET. Applying these updates will cause optimized (compiled) versions of some .NET assemblies to become invalid once the server is restarted.

In order to monitor the health of VDBs, the Delphix Engine will invoke various scripts using PowerShell on a regular basis, once for each VDB, every 5 minutes.

Because PowerShell uses Microsoft.NET assemblies, the scripts issued as part of VDB monitoring may need to dynamically recompile these assemblies each time PowerShell is run. This can result in monitoring scripts that take up to 20 times longer than normal.

Depending on the number of VDBs, and the number of CPUs available on the Windows host, this can result in Delphix-initiated PowerShell  commands consuming up to 100% of available CPU on a Windows Staging or Target host for extended periods of time.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies
Major Release All Sub Releases
6.0 6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0

5.3

5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0

5.2

5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1

5.1

5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0

5.0

5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1 ,5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

4.3

4.3.1.0, 4.3.2.0, 4.3.2.1, 4.3.3.0, 4.3.4.0, 4.3.4.1, 4.3.5.0

4.2

4.2.0.0, 4.2.0.3, 4.2.1.0, 4.2.1.1, 4.2.2.0, 4.2.2.1, 4.2.3.0, 4.2.4.0 , 4.2.5.0, 4.2.5.1

4.1

4.1.0.0, 4.1.2.0, 4.1.3.0, 4.1.3.1, 4.1.3.2, 4.1.4.0, 4.1.5.0, 4.1.6.0

Resolution

In many cases, the .NET optimization service will trigger automatically, and the issue will self-resolve within a few hours of the Windows update being applied.

Occasionally, high CPU utilization or other system configuration may prevent the .NET optimization service from running, and the invalid assemblies must be recompiled manually.

To do this, the following commands can be run on affected servers to recompile (optimize) any invalid .NET assemblies:

net stop "Delphix Connector"
C:
cd \Windows\Microsoft.NET\Framework64\v4.0.30319
ngen update
net start "Delphix Connector"

 

Note

Note:

While the Delphix Connector is stopped, the Delphix Engine will be unable to monitor the status of related dSources and VDBs. Jobs on VDBs and Staging databases will fail until the Delphix Connector service is restarted.

To prevent future recurrences of these issue, consider running ngen update manually after applying Windows Updates.

 


Related Articles

The following articles may provide more information or related information to this article: