Delphix Environment Refresh Fails with RoboCopy exit code 9 (KBA10943)
KBA
KBA#Applicable Delphix Versions
- Click here to view the versions of the Delphix engine to which this article applies
-
Date Release Mar 20, 2024 21.0.0.0 Feb 21, 2024 20.0.0.0 Jan 25, 2024 19.0.0.0 Dec 20, 2023 | Jan 10, 2024 18.0.0.0 | 18.0.0.1 Nov 21, 2023 17.0.0.0 Oct 18,2023 16.0.0.0 Sep 21, 2023 15.0.0.0 Aug 24, 2023 14.0.0.0 Jul 24, 2023 13.0.0.0 Jun 21, 2023 12.0.0.0 May 25, 2023 11.0.0.0 Apr 13, 2023 10.0.0.0 Mar 13, 2023 | Mar 20, 2023 9.0.0.0 | 9.0.0.1 Feb 13, 2023 8.0.0.0 Jan 12, 2023 7.0.0.0 Releases Prior to 2023 Major Release All Sub Releases 6.0 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0, 6.0.13.1, 6.0.14.0, 6.0.15.0, 6.0.16.0, 6.0.17.0, 6.0.17.1, 6.0.17.2
Troubleshooting Environment Refresh Failures Due to RoboCopy Exit Code 9
Environment refreshes in Delphix may fail when Delphix executes the Windows Robocopy command to copy new files that it has transferred to the host from the client-tmp folder into the target folder when the files in the target folder are in use. The Delphix Environment Refresh job will fail with an error similar to this:
Command: RoboCopy C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_b9b38a57b15f_host\client-tmp C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_b9b38a57b15f_host\client /MIR /NP /NFL /R:0 /W:0 failed with exit code: 9
There are many reasons why the files could be in use preventing RoboCopy from overwriting them:
- Antivirus programs are known to lock files. Delphix recommends excluding the Delphix Toolkit directory from antivirus scanning to prevent conflicts.
- Older versions of the Delphix Connector made excessive PowerShell calls. A bug fix went into Delphix Connector 1.27 and higher that reduces the number of PowerShell calls by about 40%. Upgrading the Delphix Connector to version 1.27 or higher can help avoid file in use problems.
- It is possible that there are long running queries or PowerShell processes that keep files locked for longer than expected. Monitor the Windows Task Manager for long running
dlpxrunas.exeprocesses. If there are any running very long, right click on them to see their properties to see what command they might be running. Restoring a very large backup might be normal but if in doubt, contact Delphix Technical Support. - Very busy staging hosts that are constantly ingesting backups from source databases. In very rare cases, it may be necessary to temporarily disable dSources to allow the environment to refresh successfully.
Prerequisites
This article combines two approaches to find what is causing the "file in use" error:
- The RoboCopy command can also be run with the
/Vverbose flag interactively if timed correctly to see which files are in use. - Microsoft's Process Monitor utility. This utility records real-time file access and allows you to set a filter to a specific file or directory to limit the output and see which processes are accessing the file.
Resolution
In this first step, RoboCopy was repeatedly executed with the /V verbose flag (adjusting the paths for each environment). The command must be executed repeatedly because the client-tmp folder is transient and gets deleted as soon as the environment refresh job fails.
C:\temp>RoboCopy "C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_b9b38a57b15f_host\client-tmp" "C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_b9b38a57b15f_host\client" /MIR /NP /NFL /R:0 /W:0 /V
-------------------------------------------------------------------------------
ROBOCOPY :: Robust File Copy for Windows
-------------------------------------------------------------------------------
Started : Friday, March 15, 2024 10:35:41 AM
Source : C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_b9b38a57b15f_host\client-tmp\
Dest : C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_b9b38a57b15f_host\client\
Files : *.*
Options : *.* /V /NFL /S /E /DCOPY:DA /COPY:DAT /PURGE /MIR /NP /R:0 /W:0
------------------------------------------------------------------------------
0 C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_b9b38a57b15f_host\client-tmp\
1 C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_b9b38a57b15f_host\client-tmp\bootstrap\
2024/03/15 10:35:41 ERROR 32 (0x00000020) Copying File C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_b9b38a57b15f_host\client-tmp\bootstrap\dlpxrunas.exe
The process cannot access the file because it is being used by another process.
1 C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_b9b38a57b15f_host\client-tmp\dsp\
0 C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_b9b38a57b15f_host\client-tmp\dsp\libs\
24 C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_b9b38a57b15f_host\client-tmp\dsp\libs\com.delphix.common\
6 C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_b9b38a57b15f_host\client-tmp\dsp\libs\com.delphix.host\
...etc...
1 C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_b9b38a57b15f_host\client-tmp\native\
2024/03/15 10:35:42 ERROR 32 (0x00000020) Copying File C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_b9b38a57b15f_host\client-tmp\native\sqljdbc_auth.dll
The process cannot access the file because it is being used by another process.
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 44 0 44 0 0 0
Files : 134 132 0 0 2 0
Bytes : 38.70 m 38.38 m 0 0 323.6 k 0
Times : 0:00:01 0:00:00 0:00:00 0:00:00
The RoboCopy ERROR 32 (0x00000020) is somewhat misleading in this instance. This error means that a file or folder cannot be overwritten because it is being used by another process. The error makes it sound like the file being copied is in use which is possible if an antivirus program had latched onto the file(s). In this particular case, there were no antivirus programs running so Microsoft's Process Monitor program was utilized to see which file was actually in use.
To see which process is accessing a file in the Delphix Connector directory, start the Process Monitor utility and add a filter that specifies the Path column and use the contains relation to specify the directory containing the file or the name of the file that is getting locked. Using the above error messages as an example, you could specify the DelphixConnector directory in the filter. Adding the filter type of Result=SHARING VIOLATION can also help narrow down the search quickly:
In this example we see that RoboCopy was able to successfully read the dlpxrunas.exe program from the client-tmp folder but encountered a SHARING VIOLATION when trying to overwrite the existing file in the toolkit directory.
The resolution in this case study was to upgrade the Delphix Connector to 1.27 or higher but since there can be many different causes, other solutions should be considered.