FSEvents on macOS
nathany opened this issue · 16 comments
Requested by @robfig at howeyc/fsnotify#54 in order to watch large trees.
To be implemented as a stand-alone repository for experimentation (#23) before determining how to incorporate it into fsnotify.
System requirements:
- OS X 10.6 or better (matching the system requirements of Go 1.3). Though some features require 10.7, or perhaps even higher.
- Likely require cgo with Xcode installed.
Prior work:
Reference:
- Advanced Mac OS X by Mark Dalrymple (@markd2) contains a chapter on FSEvents
- FSEvents Reference
- ArsTechnica
Work has begun on this as an external project that can eventually be used from fsnotify.
https://github.com/go-fsnotify/fsevents
TODO:
- Integration tests
- Refine API
- EventFlags.String()
Would it make sense to work in a monorepo for fsnotify? Bring the fsevents library into a subfolder and work on it here, even before it's integrated and used by fsnotify.
Yes, I agree that would make sense.
Some people are using fsnotify/fsevents so we'd probably leave it there and just move maintenance here. We could put a message up over there -- or not.
I'm still undecided whether it should be internal/ here initially.
I'm also not sure if we should try to move fsnotify towards this new driver system (#104), or build out something entirely new. Perhaps coming back later to retrofit fsnotify on top of an intermediary API.
Can someone explain what FSEvents is? I don't understand; afaict fsnotify already supports macOS and doesn't require cgo. Is that correct?
Hello, what's the status?
Can someone explain what FSEvents is?
According to this page there are two mechanisms for monitoring files/directories on Mac: kernel queues (kqueues), or file system events (FSEvents).
File system events are intended to provide notification of changes with directory-level granularity. For most purposes, this is sufficient. In some cases, however, you may need to receive notifications with finer granularity. For example, you might need to monitor only changes made to a single file. For that purpose, the kernel queue (kqueue) notification system is more appropriate.
If you are monitoring a large hierarchy of content, you should use file system events instead, however, because kernel queues are somewhat more complex than kernel* events, and can be more resource intensive because of the additional user-kernel communication involved.
*I assume they meant file system events here.
Ah! Makes sense. Thanks @Timmmm
I hope they will be (or currently are?) an optional part of fsnotify, so that we don't need to include cgo.
To be implemented as a stand-alone repository for experimentation (#23) before determining how to incorporate it into fsnotify.
Any timeline to use https://github.com/fsnotify/fsevents in this repo?
No timeline.
This would resolve #129 as well as issues like gohugoio/hugo#6109. Would be really nice.
We may want to do it at the same time as #114 and #21 to have recursive watching across systems.
Any idea on what the plan would be for incorporating FSEvents? Were issues found in the experimentation?
What can the community do at this point to help move this cause?
github.com/fsnotify/fsevents uses CGO, and I'd really like to avoid using CGO. The main reason for this is that it just makes building applications that depend on fsnotify harder: you will need a C compiler, and cross-compiling from e.g. Linux to macOS – which is easy now – will be very hard as well. We can use some shims and build tags to allow people to use CGO_ENABLED=0 and -tags fsnotify_no_events, but then FSEvents won't be available in that build which would be confusing.
So the path forward on this is that support for FSEvents will have to be added to x/sys/unix first, similar to how illumos/Solaris FEN support was added there before we added support here.
I don't have a macOS machine; just a (slow) QEMU VM. It's kind of hard for me to work on this, and I'm unlikely to work on it in the near future, but this is the path forward to get it properly added to fsnotify. In the meanwhile, people can use github.com/fsnotify/fsevents protected with some build tags if they want, but it's a bit awkward.
github.com/ebitengine/purego could be used to skirt the usage of cgo.
The go command now disables cgo by default on systems without a C toolchain.
https://go.dev/blog/go1.20
Falling back from FSEvents to kqueue on macOS would be confusing, that's true. It may work, but it would be confusing to the programmer or end-user compiling the software, and could make bug reports confusing too.
macOS doesn't have the C compiler installed by default, though a developer may not get very far without installing the Xcode Command Line Tools. It's been a while since I've done a fresh install, but I feel like it didn't take long after installing the Homebrew package manager before I had to get a C compiler installed too.
Cross-compiling seems like a more significant issue. Zig has been innovating in that space by bundling libc and a C/C++ compiler, but that could be a difficult proposal to sell for Go.
@arp242 Are you still without a dedicated Mac?
Are you still without a dedicated Mac?
I don't expect that'll change any time soon, if ever, and even if it did, I wouldn't be hugely interested to pick this up either to be honest. It's a comparatively big project: we're talking about a full week, maybe even two weeks of of full-time work? Even going with the existing cgo solution would need quite a bit of attention to get it up to spec.
But spending so much of my free time on something I'll never personally see the benefits of is kind of where I draw the line. That doesn't mean I wouldn't like to see support for it, or don't want to spend any time on it, but it'll have to be like the illumos/FEN stuff, where nshalman did the bulk of the work and I just helped out where I could, or something like that.