Does sysmon64 lock/remove files?
Opened this issue · 9 comments
We've been having a very strange issue with a particular Azure Devops pipeline over the last few months. We have three build agents, and the pipeline would consistently fail on two of them, and consistently work fine on the third. The failures are always the same - one specific project would fail to compile (always the same project) because it could not write the assembly object file (CSC2012: Access denied).
Analysis using Procmon showed that on the problem agents, immediately after the file is written by the compiler, subsequent access to the path would show PENDING DELETE.
We immediately suspected some heuristic rule in AV software (we run Symantec and Cylance) was quarantining the file, but after completely uninstalling both we were still seeing the error.
Then we noticed that sysmon64 was only running on the two problem agents and not on the third. After removing sysmon from one of the problem agents, the build started working fine on that agent.
Is there a way to mitigate this or do we simply have to turn off sysmon on these agents?
@darkoperator By agents, you mean Symantec, Cylance and sysmon64? We had turned off Symantec and Cylance and the problem was still occurring until we also turned off sysmon, so presumably sysmon was the sole agent at that point.
I don't know how sysmon was configured - that's owned by a separate security team, but I can find out.
I mention it because when a minifilter is set for drives only the product with the lowest altitude number will trigger, report and/or block, you can not correlate between them. In the case of Sysmon I suspect there is a file delete rule that is causing the block, that is why I asked about the config.
@darkoperator If I'm understanding the docs correctly, a file delete rule would watch for files being deleted and just log an event and/or copy the file, right? I'm looking to understand why a file is apparently being prevented from being created (or accessed after creation) by sysmon. It was my understanding that sysmon is only about monitoring and not active prevention.
I have meeting with the security team this afternoon - I'll try to get the full config file they're using.
I got their config file and it has a ton of rules, but none of them are FileDelete or FileDeleteDetected rules.
We ran some more experiments and we can consistently see that the build works fine when the sysmon64 service is stopped, and always fails to write one specific object dll file when the sysmon64 service is running.
By trial and error we traced the issue to the Event ID 7 (ImageLoad) rules. If we commented out all includes and excludes in this section the issue disappeared.
We then added an ImageLoad exclude rule for devenv.exe and the problem also went away (with all of the other rules restored).
I did post on Microsoft Q&A under the [windows-sysinternals-sysmon] tag: (https://docs.microsoft.com/answers/topics/164854/windows-sysinternals-sysmon.html)
Is there a more suitable spot?