Skip to main content
Delphix

SQL Server VDB Operations May Be Slowed By Windows CustomDestinations Directory (KBA4557)

 

 

KBA

KBA #4557

Issue

When running Delphix Engine 6.0.8.1 or earlier, SQL Server VDB operations may perform more slowly on Target servers that have been in operation for several months or years, running Windows Server 2008R2, 2012 or 2012R2.

This is triggered by a side-effect of the native Windows "Recent Files" behavior, when large numbers of PowerShell operations are being run concurrently.

This may include the following symptoms:

  • Large numbers of PowerShell processes running in parallel, for more than 10 seconds each
  • Very high CPU usage by PowerShell processes
  • In some circumstances, VDB operations that fail with "Connection Reset" or "Timeout" errors

This is most pronounced in target environments that have been running for many months, with dozens of active VDBs, especially if multiple Delphix Engines are in use.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies
Major Release All Sub Releases
6.0 6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2. 6.0.5.0, 6.0.6.0, 6.0.6.1,. 6.0.7.0, 6.0.8.0, 6.0.8.1

5.3

5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0

5.2

5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1

5.1

5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0

5.0

5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1 ,5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

4.3

4.3.1.0, 4.3.2.0, 4.3.2.1, 4.3.3.0, 4.3.4.0, 4.3.4.1, 4.3.5.0

4.2

4.2.0.0, 4.2.0.3, 4.2.1.0, 4.2.1.1, 4.2.2.0, 4.2.2.1, 4.2.3.0, 4.2.4.0 , 4.2.5.0, 4.2.5.1

4.1

4.1.0.0, 4.1.2.0, 4.1.3.0, 4.1.3.1, 4.1.3.2, 4.1.4.0, 4.1.5.0, 4.1.6.0

Diagnosis

To confirm whether this issue is specifically related to the Recent Files functionality, navigate to the following directory on the affected Windows host:

C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\Windows\Recent

 

tip

Tip:

It will be fastest to paste this location directly into the Address Bar of a File Explorer window. The "AppData" folder is usually hidden on Windows hosts, and may not be visible without the "Show Hidden Files and Folders" option enabled.

Right-click the "CustomDestinations" folder and select Properties. It may take some time for the dialog to continue counting and settle on a final number.

clipboard_e190130c00e0cbd120c64f84c0a1c9e09.png

Servers showing more than 100,000 files in this directory are likely to experience degraded performance of VDB operations.

Resolution

To resolve this issue for all Environments connected to the Delphix Engine, please upgrade to version 6.0.9.0 or later.

Delphix Engine version 6.0.9.0 and later will automatically attempt to mark the "jump list" file for PowerShell as Read Only. This should resolve immediate symptoms and prevent any additional files from being created in the CustomDestinations directory.

If the Delphix Engine is unable to make this change or the file cannot be located, a Fault will be raised containing troubleshooting information.

To resolve this in earlier versions of the Delphix Engine, complete the following changes:

  • Marking the file 590aee7bdd69b59b.customDestinations-ms as read-only, preventing PowerShell from modifying its jump list file. The file is located in the folder C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\Windows\Recent\CustomDestinations
  • Migrating VDBs to a newer Target Environment running Windows Server 2016 or newer (check the Support Matrix for your version of the Delphix Engine to ensure that it is supported)

Marking the file 590aee7bdd69b59b.customDestinations-ms as read-only will not remove any existing .TMP files from the CustomDestinations directory You may want to take that step as well.

Explanation

When PowerShell is launched in Windows 2012R2 or earlier, it attempts to modify its Jump List file, which is located in the following directory:

C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\Windows\Recent\CustomDestinations

This is native Windows and PowerShell behavior, and is not specific to scripts launched by the Delphix Engine.

When multiple PowerShell processes are being launched in parallel, file access or collision errors may occur during this step. This does not impact operation of the scripts, but will leave temporary files in the CustomDestinations directory, with file names such as 590aee7bdd69b59b.customDestinations-ms~RF4dd6039.TMP.

This has been observed on some Delphix Engine target hosts, especially in cases where there are dozens of VDBs being managed and monitored by multiple Delphix Engines.

By default, Windows also assigns "short file names" to each of these temporary files, such as 590aee~4.TMP.

As the number of files in the CustomDestinations directory increases, the amount of work required to generate a new unique file name increases, and all PowerShell operations invoked by the Delphix Connector are slowed down.

In extreme cases, this may cause PowerShell operations that normally complete in a few seconds to take a minute or longer. This may be further slowed down by Anti-Virus or Endpoint Protection software.