Several problem on OS X

[elgs]

1, if I drag 5 files to the monitored folder, each ~170MB, only the first 3 trigger the CREATE event;
2, if the monitored files are deleted, but are still in the Trash, even if I copy the same files again to the monitored the folder, the CREATE events are not triggered;

[nathany]

@elgs This is very odd indeed. Would you mind sharing some of the code you use to setup a watcher and process events? I'll try to help reproduce the issue.

Please double check that you have the latest code for fsnotify go get -u github.com/howeyc/fsnotify and also let me know which version of OS X you're on. Thanks!

[elgs]

I did some more test, it works without any problem. And I think I know my problem now. In my code, whenever a directory is created in the watched directory, I add the new directory to watch. If the files inside the new directory is created before the new directory is added to watch, the new files within the new directory will not trigger the CREATE events. It looks like when a new directory is added to watch, I have to scan inside the new directory, because there is no guarantee that the new directory is watched before the files inside are copied.

[elgs]

I can reproduce this problem with the following code:

package main

import (
    "log"
    "github.com/howeyc/fsnotify"
)

func main() {
    watcher, err := fsnotify.NewWatcher()
    if err != nil {
        log.Fatal(err)
    }

    done := make(chan bool)

    go func() {
        for {
            select {
            case ev := <-watcher.Event:
                log.Println("event:", ev)
            case err := <-watcher.Error:
                log.Println("error:", err)
            }
        }
    }()

    err = watcher.Watch("/Volumes/User/Home/Desktop/a")
    if err != nil {
        log.Fatal(err)
    }

    <-done
}

In the a directory, when I run touch a s d f g h k l, it outputs:

2014/02/02 03:31:00 event: "/Volumes/User/Home/Desktop/a/a": CREATE
2014/02/02 03:31:00 event: "/Volumes/User/Home/Desktop/a/d": CREATE
2014/02/02 03:31:00 event: "/Volumes/User/Home/Desktop/a/s": CREATE
2014/02/02 03:31:00 event: "/Volumes/User/Home/Desktop/a/l": CREATE

This happens only on OS X, on Linux, it works as expected.

I'm not sure if this is a bug from this library or from the operating system. I used to use the file watcher in Java7, the Mac version is also less responsive than Linux and Windows. Seems the Mac version is using polling, where as Linux and Windows are using callback. I'm not quite sure about the details.

[nathany]

Thanks for the example code. I can reproduce it on OS X 10.9. The implementation differs from one OS to the next, which is why you aren't seeing this issue on Linux.

It seems to be related to sendDirectoryChangeEvents in fsnotify_bsd.go. It is actually doing a ReadDir and scanning for created files. However, it someone gets confused as to which files already exist:

/Users/nathany/Desktop/a/a false
/Users/nathany/Desktop/a/s false
2014/02/01 22:26:28 event: "/Users/nathany/Desktop/a/a": CREATE!
2014/02/01 22:26:28 event: "/Users/nathany/Desktop/a/s": CREATE!
/Users/nathany/Desktop/a/a true
/Users/nathany/Desktop/a/d true
/Users/nathany/Desktop/a/f true
/Users/nathany/Desktop/a/g true
/Users/nathany/Desktop/a/h true
/Users/nathany/Desktop/a/j true
/Users/nathany/Desktop/a/k true
/Users/nathany/Desktop/a/l false
/Users/nathany/Desktop/a/s true
2014/02/01 22:26:28 event: "/Users/nathany/Desktop/a/l": CREATE!

I'm still investigating as to why.

[nathany]

@howeyc I booted up my BSD Vagrant box and it works perfectly there. Just a problem on OS X.

[nathany]

So the problem is that sendDirectoryChangeEvents does a ReadDir and sends a create event for a file. Then it calls watchDirectoryFiles which does a ReadDir as well. By this time more files have been touched so ReadDir gets a different result. watchDirectoryFiles is marking those files as existing without sending create events. And since they have been flagged as existing those files are never sent as create events the next time sendDirectoryChangeEvents is called.

I'm not sure exactly what the fix is yet, because watchDirectoryFiles is called from one other place, and I'm still getting to grips with this code. @howeyc Now that the problem is known, is the solution obvious to you?

[elgs]

I found the same problem on Linux, at least once. When I touch a lot of new files, one is missing.

$ touch a b c d e f

b is missed out, this time. It seems to be much less likely to happen on Linux than it on OS X.

[nathany]

@elgs The issue with missing create events on OS X has been resolved (thanks to @zhsso) in v1.0.2 with a backport to v0.9.2. Please see https://github.com/go-fsnotify/fsnotify.

[Deleted user]

This issue is still marked as open?

[nathany]

@corydoras. Most of the work on fsnotify is over at https://github.com/fsnotify/fsnotify

That said, I'm not sure if all these issues have been addressed yet over there.

[elgs]

I'm sorry for a late reply. I will check it and update here.