Skip to main content
Delphix

SQL Server VDB Operations May Be Slowed By Windows CustomDestinations Directory (KBA4557)

 

Issue

In some environments, SQL Server VDB operations may begin to perform more slowly on Target servers that have been in operation for several months or years, running Windows Server 2008R2, 2012 or 2012R2.

This is triggered by a side-effect of the native Windows "Recent Files" behavior, when large numbers of PowerShell operations are being run concurrently.

This may include the following symptoms:

  • Large numbers of PowerShell processes running in parallel, for more than 10 seconds each
  • Very high CPU usage by PowerShell processes
  • In some circumstances, VDB operations that fail with "Connection Reset" or "Timeout" errors

This is most pronounced in target environments that have been running for many months, with dozens of active VDBs, especially if multiple Delphix Engines are in use.

Applicable Delphix Versions

This article applies to the following versions of the Delphix Engine:

Major Release

All Sub Releases

All versions All versions

Diagnosis

To confirm whether this issue is specifically related to the Recent Files functionality, navigate to the following directory on the affected Windows host:

C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\Windows\Recent

Note: It will be fastest to paste this location directly into the Address Bar of a File Explorer window. The "AppData" folder is usually hidden on Windows hosts, and may not be visible without the "Show Hidden Files and Folders" option enabled.

Right-click the "CustomDestinations" folder and select Properties. It may take some time for the dialog to continue counting, and settle on a final number.

clipboard_e190130c00e0cbd120c64f84c0a1c9e09.png

Servers which show more than 100,000 files in this directory are likely to experience degraded performance of VDB operations.

Resolution

To resolve slow VDB operations caused by files in the CustomDestinations directory, work with your System Administrators to clean up the directory. This behavior is related to the operation of Windows, and is not specific to the Delphix Engine.

Changes that may offer relief from the symptoms described in this article include:

  • Migrating VDBs to a newer Target Environment running Windows Server 2016 (Delphix Engine >= 5.2.3) or Windows Server 2019 (Delphix Engine >= 5.3.3)
  • Deleting all .TMP files from the folder C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\Windows\Recent\CustomDestinations
  • Renaming the folder C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\Windows\Recent\CustomDestinations, so that Windows automatically re-creates a new, smaller folder
  • Disabling Windows Short File Name generation for C:\, using fsutil 8dot3name
  • Marking the file 590aee7bdd69b59b.customDestinations-ms as read-only, preventing PowerShell from modifying its jump list file. The file is located in the CustomDestinations folder

These changes may not be appropriate for all environments, and should be discussed with your System Administrators or raised with Microsoft for further evaluation before action is taken.


Explanation

When PowerShell is launched in Windows 2012R2 or earlier, it attempts to modify its Jump List file, which is located in the following directory:

C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\Windows\Recent\CustomDestinations

This is native Windows and PowerShell behavior, and is not specific to scripts launched by the Delphix Engine.

When multiple PowerShell processes are being launched in parallel, file access or collision errors may occur during this step. This does not impact operation of the scripts, but will leave temporary files in the CustomDestinations directory, with file names such as 590aee7bdd69b59b.customDestinations-ms~RF4dd6039.TMP.

This has been observed on some Delphix Engine target hosts, especially in cases where there are dozens of VDBs being managed and monitored by multiple Delphix Engines.

By default, Windows also assigns "short file names" to each of these temporary files, such as 590aee~4.TMP.

As the number of files in the CustomDestinations directory increases, the amount of work required to generate a new unique file name increases, and all PowerShell operations invoked by the Delphix Connector are slowed down.

In extreme cases, this may cause PowerShell operations that normally complete in a few seconds to take a minute or longer. This may be further slowed down by Anti-Virus or Endpoint Protection software.