Does fsevents_monitor can capture all file updates ?

[bric3]

Hi,

Thanks for this utility, this issue is more a question about fs events on macos than to report a problem, so when watching a file directly, I noticed that fs monitor don't report events on a log file, I'm using JetBrains IntelliJ IDEA log files as an example.

$ fswatch -x --monitor=fsevents_monitor /Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log
^C%

$ fswatch -x --monitor=kqueue_monitor /Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log PlatformSpecific
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log Updated
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log Updated PlatformSpecific
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log PlatformSpecific
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log Updated
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log Updated PlatformSpecific
^C%

It seems that only kqueue_monitor is picking the events on this log file.

However when I use some command lines to modify the file (additionnally to IntelliJ IDEA), then fsevents_monitor picks up some events, altthough they are not the same as those reported by kqueue_monitor :

$ fswatch -x --monitor=fsevents_monitor /Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log PlatformSpecific IsFile <=== touch idea.log
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log Updated IsFile <=== echo "line" > idea.log

Also, I tried to use fswatch to watch events on the parent directory:

$ fswatch -xr --monitor=fsevents_monitor /Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log PlatformSpecific IsFile <=== touch idea.log
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log Updated IsFile <=== echo "line" > idea.log

FS events monitor picks events the same events when touching or appending via a command line tool (touch or echo "" >> ), but not from the process (IntelliJ IDEA) that is really appending log messages.

If using kqueue monitor for this folder, nothing gets captured unless I use the -r option, not even when the file modified with a command line tool (touch or echo "" >> ).

$ fswatch -xr --monitor=kqueue_monitor /Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log Updated PlatformSpecific
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log PlatformSpecific
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log Updated
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log Updated PlatformSpecific
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log Updated PlatformSpecific
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log Updated PlatformSpecific
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log PlatformSpecific
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log Updated
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log PlatformSpecific
/Users/bric3/Library/Logs/JetBrains/IntelliJIdea2020.3/idea.log Updated

The documentation (1.14.0) suggests that fsevents is to be preferred on OSX, but is it really the prefered way if monitoring specific files. Or am I missing something ?

Thanks in advance for the feedback !

[emcrisostomo]

Hi @bric3, I don't know, I'll try to reproduce it and investigate it

[bric3]

@emcrisostomo Hi, did you had the chance to investigate ?

[alvis]

I notice the same issue when I try to monitor node_modules folders created by pnpm using fsevent monitor.
Seems like when the number of file changes is small, it's working fine.
But whenever there's a large number of file changes, some are missing RANDOMLY.

The probabilistic nature of the issue may suggest it's some kind of buffer or queue size overflow?

@emcrisostomo is there any queue size parameter we can tweak?