Troubleshooting File Access Issues Caused By Windows Defender (KBA10458)
KBA
KBA#Applicable Delphix Versions
- Click here to view the versions of the Delphix engine to which this article applies
-
Date Release Sep 21, 2023 15.0.0.0 Aug 24, 2023 14.0.0.0 Jul 24, 2023 13.0.0.0 Jun 21, 2023 12.0.0.0 May 25, 2023 11.0.0.0 Apr 13, 2023 10.0.0.0 Mar 13, 2023 | Mar 20, 2023 9.0.0.0 | 9.0.0.1 Feb 13, 2023 8.0.0.0 Jan 12, 2023 7.0.0.0 Releases Prior to 2023 Major Release All Sub Releases 6.0 6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0, 6.0.13.1, 6.0.14.0, 6.0.15.0, 6.0.16.0, 6.0.17.0, 6.0.17.1, 6.0.17.2 5.3
5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0, 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0 5.2
5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1
5.1
5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0
5.0
5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1, 5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4
Troubleshooting File Access Issues
Antivirus programs like Windows Defender lock files while they are scanning them. While the files are locked by the antivirus program, Delphix will be prevented from writing to existing files and/or creating new files.
This issue can come up at different times. During an environment discovery or refresh job, the Delphix Engine starts writing large files by sending chunks. If the antivirus program locks the file before Delphix finishes sending the chunks, it can cause the environment refresh to fail with an error "The process cannot access the file because it is being used by another process"):
[2023-05-10T13:48:39,247][INFO][ccc.impl.connectorconnection.C3BaseConnectorConnectionImpl#lambda$processWriteFile$5:494][pool-434673-thread-1][] [windows2019.acme.com:9100/5ea4de13-3049-4bbd-8591-f209d73ce20e]: writefile failure: UNSUCCESSFUL: java.io.FileNotFoundException: C:\Program Files\Delphix\DelphixConnector\Delphix_COMMON_ff790349fb6c_host\tmp\jdk.zip (The process cannot access the file because it is being used by another process)
The issue can present itself at other times, including when Delphix is rolling the Delphix Connector's log files. If the antivirus program locks the log files while Delphix is trying to manage them, an alert will be raised indicating that the operating system prevented access to the file with an "Access is denied" error:
Error: "exception.host.client.launch.failed {192.168.0.23|2023-09-22 14:05:15,775 main ERROR RollingFileManager (E:\MSSQL\Data\DelphixConnector\Delphix_446155368543_6af91945da05_40_host\log\connector/trace.log) java.io.FileNotFoundException: E:\MSSQL\Data\DelphixConnector\Delphix_446155368543_6af91945da05_40_host\log\connector\trace.log (Access is denied) java.io.FileNotFoundException: E:\MSSQL\Data\DelphixConnector\Delphix_446155368543_6af91945da05_40_host\log\connector\trace.log (Access is denied)
This issue isn't limited to Windows Defender but this article focuses on commands provided by Windows Defender to avoid the issue.
Delphix generally recommends that the Delphix Connector directory be excluded from anti-virus scanning ( Best practices for target DB and OS settings ):
Anti-virus programs can impact both performance and operation. Delphix recommends anti-virus scanning exclude folders where Delphix files are maintained, in addition to the normal exclusions put in place for MSSQL operation.
Microsoft also recommends excluding SQL Server related directories ( Configure antivirus software to work with SQL Server ). See the section titled "Directories and file name extensions to exclude from virus scanning". Since the Delphix database related files live under the connector directory, this directory is also recommended to be excluded ( C:\Program Files\Delphix\DelphixConnector ).
Prerequisites
This article utilizes Microsoft's Process Monitor utility. This utility records real-time file access and allows you to set a filter to a specific file or directory to limit the output and see which processes are accessing the file.
Resolution
To see which process is accessing a file in the Delphix Connector directory, start the Process Monitor utility and add a filter that specifies the Path column and use the contains relation to specify the directory containing the file or the name of the file that is getting locked. Using the above error messages as an example, you could specify the DelphixConnector directory or jdk.zip or access.log.
This article shows how to create a dummy file in the directory and filter on that file name (delphixtest in the below image):
Once the Process Monitor trace is running, create a new file in the DelphixConnector subdirectory. Using PowerShell, you can create an empty file:
Out-File -FilePath 'C:\Program Files\Delphix\DelphixConnector\DelphixTest.log'
If Windows Defender (or another antivirus program) is actively scanning this directory, as soon as PowerShell closes the file, you can see the Windows Defender MsSense.exe executable start to access the file:
There is a PowerShell command that can be utilized to list directories that are excluded but the output may be long and difficult to parse:
Get-MpPreference | Select-Object -Property ExclusionPath -ExpandProperty ExclusionPath
Microsoft provides a utility that lets you more precisely check whether or not a path is excluded ( Configure and manage Microsoft Defender Antivirus with the mpcmdrun.exe command-line tool ).
See the section titled "Validate the exclusion list by using MpCmdRun":
Start, CMD (Run as admin)
cd "%programdata%\microsoft\windows defender\platform"
cd 4.18.2111-5.0 (Where 4.18.2111-5.0 is this month's Microsoft Defender Antivirus "Platform Update".)
MpCmdRun.exe -CheckExclusion -path <path>
There may be several directories there so you would want to choose the most recent:
Directory of C:\ProgramData\Microsoft\Windows Defender\Platform
08/10/2023 08:22 PM <DIR> 4.18.23070.1004-0
08/31/2023 10:32 AM <DIR> 4.18.23080.2006-0
10/11/2023 02:25 PM <DIR> ..
10/11/2023 02:25 PM <DIR> .
10/11/2023 02:25 PM <DIR> 4.18.23090.2008-0
Using this utility as an "Administrator", here are two examples. One illustrates a directory that has been excluded and the other that is not excluded:
C:\ProgramData\Microsoft\Windows Defender\Platform\4.18.23090.2008-0>MpCmdRun.exe -CheckExclusion -path "C:\Program Files\Delphix\DelphixConnector"
C:\Program Files\Delphix\DelphixConnector [\Device\HarddiskVolume4\Program Files\Delphix\DelphixConnector] is excluded. Exit code is 0.
C:\ProgramData\Microsoft\Windows Defender\Platform\4.18.23090.2008-0>MpCmdRun.exe -CheckExclusion -path "C:\temp"
C:\temp [\Device\HarddiskVolume4\temp] is not excluded. Exit code is 1.
To resolve the issue, add an exclusion for the Delphix Connector directory. You can use the Windows Defender menus or a simple PowerShell command executed from an "Administrator" account. Be sure to specify the correct path to your Delphix Connector installation:
Add-MpPreference -ExclusionPath 'C:\Program Files\Delphix\DelphixConnector'