In some environments, SQL Server VDB operations may begin to perform more slowly on Target servers that have been in operation for several months or years, running Windows Server 2008R2, 2012 or 2012R2.
This is triggered by a side-effect of the native Windows "Recent Files" behavior, when large numbers of PowerShell operations are being run concurrently.
This may include the following symptoms:
- Large numbers of PowerShell processes running in parallel, for more than 10 seconds each
- Very high CPU usage by PowerShell processes
- In some circumstances, VDB operations that fail with "Connection Reset" or "Timeout" errors
This is most pronounced in target environments that have been running for many months, with dozens of active VDBs, especially if multiple Delphix Engines are in use.
Applicable Delphix Versions
This article applies to the following versions of the Delphix Engine:
All Sub Releases
|All versions||All versions|
To confirm whether this issue is specifically related to the Recent Files functionality, navigate to the following directory on the affected Windows host:
Note: It will be fastest to paste this location directly into the Address Bar of a File Explorer window. The "AppData" folder is usually hidden on Windows hosts, and may not be visible without the "Show Hidden Files and Folders" option enabled.
Right-click the "CustomDestinations" folder and select Properties. It may take some time for the dialog to continue counting, and settle on a final number.
Servers which show more than 100,000 files in this directory are likely to experience degraded performance of VDB operations.
To resolve slow VDB operations caused by files in the CustomDestinations directory, work with your System Administrators to clean up the directory. This behavior is related to the operation of Windows, and is not specific to the Delphix Engine.
Changes that may offer relief from the symptoms described in this article include:
- Migrating VDBs to a newer Target Environment running Windows Server 2016 (Delphix Engine >= 5.2.3) or Windows Server 2019 (Delphix Engine >= 5.3.3)
- Deleting all .TMP files from the folder
- Renaming the folder
C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\Windows\Recent\CustomDestinations, so that Windows automatically re-creates a new, smaller folder
- Disabling Windows Short File Name generation for C:\, using
- Marking the file
590aee7bdd69b59b.customDestinations-msas read-only, preventing PowerShell from modifying its jump list file. The file is located in the CustomDestinations folder
These changes may not be appropriate for all environments, and should be discussed with your System Administrators or raised with Microsoft for further evaluation before action is taken.
When PowerShell is launched in Windows 2012R2 or earlier, it attempts to modify its Jump List file, which is located in the following directory:
This is native Windows and PowerShell behavior, and is not specific to scripts launched by the Delphix Engine.
When multiple PowerShell processes are being launched in parallel, file access or collision errors may occur during this step. This does not impact operation of the scripts, but will leave temporary files in the CustomDestinations directory, with file names such as 590aee7bdd69b59b.customDestinations-ms~RF4dd6039.TMP.
This has been observed on some Delphix Engine target hosts, especially in cases where there are dozens of VDBs being managed and monitored by multiple Delphix Engines.
By default, Windows also assigns "short file names" to each of these temporary files, such as 590aee~4.TMP.
As the number of files in the CustomDestinations directory increases, the amount of work required to generate a new unique file name increases, and all PowerShell operations invoked by the Delphix Connector are slowed down.
In extreme cases, this may cause PowerShell operations that normally complete in a few seconds to take a minute or longer. This may be further slowed down by Anti-Virus or Endpoint Protection software.