Gradually slower responsiveness to change events on Windows

Question

Gradually slower responsiveness to change events on Windows

Closed this issue 5 years ago · 32 comments

I use it with gulp and have some time leaking. Looks like the problem in globs.
Also on a bigger project, where I found this problem, for nested files creates nested.css-jds43J.TMP like files for a few miliseconds. Saw it in sublime sidebar

var gulp = require('gulp');
var chokidar = require('chokidar');

gulp.task('build', function () {
    return gulp.src('src/main.css')
        .pipe(gulp.dest('dist'));
});

gulp.task('dev', function () {
    chokidar.watch('src/**/*.css')
    .on('change', function () {
        gulp.start('build');
    })
})

File structure

src/main.css
src/include/nested.css

Log

[13:07:10] Starting 'dev'...
[13:07:10] Finished 'dev' after 4.38 ms
// A few changes in a row and periodical changes in main.css
[13:07:11] Starting 'build'...
[13:07:11] Finished 'build' after 54 ms
[13:07:11] Starting 'build'...
[13:07:11] Finished 'build' after 27 ms
[13:07:11] Starting 'build'...
[13:07:11] Finished 'build' after 19 ms
[13:07:12] Starting 'build'...
[13:07:12] Finished 'build' after 17 ms
[13:07:12] Starting 'build'...
[13:07:12] Finished 'build' after 23 ms
[13:07:14] Starting 'build'...
[13:07:14] Finished 'build' after 26 ms
[13:07:14] Starting 'build'...
[13:07:14] Finished 'build' after 10 ms
[13:07:14] Starting 'build'...
[13:07:14] Finished 'build' after 19 ms
// Periodical changes in nested.css
[13:07:17] Starting 'build'...
[13:07:18] Finished 'build' after 295 ms
[13:07:19] Starting 'build'...
[13:07:19] Finished 'build' after 321 ms
[13:07:20] Starting 'build'...
[13:07:20] Finished 'build' after 305 ms
[13:07:21] Starting 'build'...
[13:07:22] Finished 'build' after 341 ms
[13:07:24] Starting 'build'...
[13:07:24] Finished 'build' after 387 ms
[13:07:26] Starting 'build'...
[13:07:26] Finished 'build' after 336 ms
[13:07:29] Starting 'build'...
[13:07:29] Finished 'build' after 361 ms
[13:07:31] Starting 'build'...
[13:07:31] Finished 'build' after 397 ms
[13:07:34] Starting 'build'...
[13:07:34] Finished 'build' after 451 ms
[13:07:36] Starting 'build'...
[13:07:37] Finished 'build' after 368 ms
[13:07:38] Starting 'build'...
[13:07:38] Finished 'build' after 425 ms
[13:07:40] Starting 'build'...
[13:07:41] Finished 'build' after 464 ms
[13:07:43] Starting 'build'...
[13:07:43] Finished 'build' after 497 ms
[13:07:45] Starting 'build'...
[13:07:45] Finished 'build' after 429 ms
[13:07:47] Starting 'build'...
[13:07:47] Finished 'build' after 491 ms
[13:07:49] Starting 'build'...
[13:07:49] Finished 'build' after 540 ms
// A few changes in a row in nested.css
[13:07:52] Starting 'build'...
[13:07:55] Finished 'build' after 3.04 s
// Again periodical changes
[13:15:57] Starting 'build'...
[13:15:58] Finished 'build' after 669 ms
[13:15:59] Starting 'build'...
[13:16:00] Finished 'build' after 681 ms
[13:16:03] Starting 'build'...
[13:16:03] Finished 'build' after 725 ms
// Changes in main.css
[13:16:06] Starting 'build'...
[13:16:06] Finished 'build' after 23 ms
[13:16:07] Starting 'build'...
[13:16:07] Finished 'build' after 9.2 ms
[13:16:07] Starting 'build'...
[13:16:07] Finished 'build' after 17 ms

Answer 1 · 2015-08-05T10:18:46.000Z

Tried gulp.watch. It works fine with exception it is gaze :).

Answer 2 · 2015-08-05T11:32:22.000Z

Chokidar isn't creating the temp files, something else is.

Based on your log I don't see why you'd be blaming chokidar. The increased times are happening between chokidar emitting a change event and the completion of build.

It looks like you've oversimplified your gulpfile example, as something would need to be processing your @imports rather than just copying main.css as-is each time. I suspect the answer to this is somewhere in the parts you've left out. What you're showing isn't really that time is leaking, just that build takes a lot longer whenever you change your nested file.

I don't know why you've observed a difference with with gulp.watch. Maybe rather than using chokidar directly you could try the gulp-watch plugin.

Answer 3 · 2015-08-05T11:47:31.000Z

@es128 I didn't simplify it. I just removed everything that do not affect on my problem. Will make a repo to reproduce it.

Answer 4 · 2015-08-05T11:50:08.000Z

What's the point of watching all css files and then just copying main.css from src to dest? Where do the nested css files come into play?

Answer 5 · 2015-08-05T11:52:45.000Z

My builder uses postcss-import for this. And with it and without I have the same problem.

Answer 6 · 2015-08-05T11:55:37.000Z

Ok, but nonetheless how is chokidar involved in increased times between starting build and finishing it? By the time you start, chokidar has already emitted its change event.

Answer 7 · 2015-08-05T11:58:08.000Z

Also this problem only on windows.

Answer 8 · 2015-08-05T11:58:19.000Z

Repo
https://github.com/TrySound/test-gulp-chokidar

Answer 9 · 2015-08-05T19:32:06.000Z

Hm.. on my home computer everything is fine. Something wrong with that machine

Answer 10 · 2015-10-13T21:16:55.000Z

@es128 https://github.com/TrySound/chokidar-leaking-test
I reproduced the problem on my win8 and win7-vm. Also my colleague has the same problem with win7.

Answer 11 · 2015-10-13T21:30:39.000Z

Ok, but nonetheless how is chokidar involved in increased times between starting build and finishing it? By the time you start, chokidar has already emitted its change event.

Your evidence indicates slowness in builds, not in watching and responsiveness of change events.

You made a comment in your repo that you ruled out gulp as the culprit, but you didn't explain how. If you want me to investigate this, please provide evidence that uses only chokidar.

Answer 12 · 2015-10-13T21:31:49.000Z

Btw try gulp.watch with latest gulp 4, which uses chokidar

Answer 13 · 2015-10-13T22:38:00.000Z

@es128 I saw chokidar in gulp.watch. It's just chokidar. Gulp doesn't matter. When I use it without gulp i got the same brake. I just don't know how to catch keypress in background and compare it with watch end. I added index.js into repo with only chokidar and hrtime to try. Wihout gulp speed decreases slower, but still decreases.

Answer 14 · 2015-10-13T22:39:52.000Z

@es128 The problem is only with **/*.* globs. *.* works well.

Answer 15 · 2015-10-13T22:41:15.000Z

@es128 It's real problem and gulp using is the most bright example.

Answer 16 · 2015-10-14T13:42:55.000Z

This still isn't described or demonstrated well enough for there to be anything actionable yet, but ok reopening. Let's keep trying to narrow down what's going on.

I just don't know how to catch keypress in background and compare it with watch end

fs.writeFile. To better demonstrate this problem, you should create a script that writes to a file on an interval and benchmark the time to detect the event.

Does usePolling: true have any impact on this phenomenon?

Answer 17 · 2015-10-14T15:53:18.000Z

@es128 Yay! With usePolling works fine!)

Answer 18 · 2015-10-14T16:00:57.000Z

@es128 Can this option negatively affect watching on *nix systems?

Answer 19 · 2015-10-14T16:10:54.000Z

Polling will be slower and consume more CPU - especially if you're watching a lot of files.

Chokidar's algorithms for handling polling vs non-polling on Windows are nearly identical, so it's starting to point toward being a bug in fs.watch on Windows or the ReadDirectoryChangesW system API it uses. Or perhaps some combination of using those and doing readdir calls. I don't know why the glob pattern would make a difference though.

If you can get to the point of reliably demonstrating the issue, we could see if it could still be demonstrated using fs.watch directly instead of chokidar. And if that proves out, report it to node.

Answer 20 · 2015-12-02T09:59:19.000Z

I think I have the same issue here: #405

Answer 21 · 2017-01-09T15:55:15.000Z

I'm getting this same problem with Gulp 4 using the following gulp file.

let gulp = require('gulp');
let rename = require('gulp-rename');

function styles() {
    return gulp.src('src/**/*.scss')
        //Issue happens with or without this rename plugin
        .pipe(rename({extname: '.test'}))
        //Issue only happens if destination directory is same as source directory
        .pipe(gulp.dest('src'));
}

gulp.task('watch', function() {
    //only happens if there is a glob here
    gulp.watch('src/**/test.html', styles);
});

Running gulp watch outputs:

If I change the watch statement to use polling the issue doesn't occur

gulp.watch('src/**/test.html', {usePolling: true}, styles);

Attached is a minimal app that reproduces the issue
gulp-issue.zip

I am using the following versions
node: v7.2.0
npm: 3.10.9
OS: Windows 10
Gulp CLI: 1.2.2
Local Gulp: 4.0.0-alpha.2

Answer 22 · 2017-02-04T11:36:35.000Z

help please ((
gulpjs/gulp#1903 # @

Answer 23 · 2017-11-01T22:01:40.000Z

This is a pretty old issue, but I've noticed that I'm seeing the same problem. Did anyone here ever find a solution other than switching to polling?

Answer 24 · 2017-11-01T23:27:27.000Z

Hey @JustMaier - I did a little bit more and switched OS 😂

Anyway... in the last year or so chokidar was quite stable on Windows machines I used.
Perhaps there's something to do with improvements in the core of Node?

Answer 25 · 2017-11-02T16:44:57.000Z

Thanks for the great idea @gustavohenke.

I was on node 6.6.2 and I updated to 8.9.0 and now it works great.

If anyone stumbles across this, update your version of node!

Answer 26 · 2017-11-03T11:33:37.000Z

@JustMaier, tried node 8.9.0 - same result for me. Guess we stuck with polling.

Answer 27 · 2017-11-06T16:10:44.000Z

@anwerso, sorry to hear that. What version of chokidar are you running? I'm just wondering what the difference is between our team setup and yours.

Here's what we're running:

OS: Windows 10 15063.674
Node: 8.9.0
Gulp: 4 Alpha 2
Chokidar: 1.4.3

Answer 28 · 2017-11-08T08:35:10.000Z

@JustMaier, I use the latest Gulp from the repo:

OS: Windows 10 15063.674
Node: 8.9.0
Gulp: 4 Alpha 2
Chokidar: 1.7.0

Answer 29 · 2017-11-08T08:49:44.000Z

@JustMaier, tried to downgrade chokidar, same results. It seems the problem somewhere else, probably in the task config.

Answer 30 · 2017-11-18T03:52:09.000Z

@anwerso Looks like I was incorrect. It seems to still be a problem with files that were pulled into the pipe via globbing (**/*.js)

What's odd is that in my case, things only slow down if there are globs in the gulp src. If there are only globs on the watch list then everything works fine.

Answer 31 · 2019-01-22T14:37:22.000Z

Still the case with node 8.11.4
Using usePolling is intead fixing this issue but dont know if this is a good thing to do.

As the Gulp/watch documentation is saying that

usePolling | boolean | false | When false, the watcher will use fs.watch() (or fsevents on Mac) for watching. If true, use fs.watchFile() polling instead - needed for successfully watching files over a network or other non-standard situations. Overrides the useFsEvents default.This option is passed directly to chokidar.

And the Chokidar documentation also

usePolling (default: false). Whether to use fs.watchFile (backed by polling), or fs.watch. If polling leads to high CPU utilization, consider setting this to false. It is typically necessary to set this to true to successfully watch files over a network, and it may be necessary to successfully watch files in other non-standard situations. Setting to true explicitly on OS X overrides the useFsEvents default. You may also set the CHOKIDAR_USEPOLLING env variable to true (1) or false (0) in order to override this option.

I came to the conclusion that usePolling is set to true when your files are on a shared network drive or your files are in an other non-standard situations.
I guess i am in the non standard situations category but i don't know why.

Is it because my source files and dest files are in the same root folder ? (sorry can't paste my gulpfile here)

But the gulpfile written above by @robianmcd is quite similar to mine because the source files and dest files are in the same root folder.

let rename = require('gulp-rename');

function styles() {
    return gulp.src('src/**/*.scss')
        //Issue happens with or without this rename plugin
        .pipe(rename({extname: '.test'}))
        //Issue only happens if destination directory is same as source directory
        .pipe(gulp.dest('src'));
}

gulp.task('watch', function() {
    //only happens if there is a glob here
    gulp.watch('src/**/test.html', styles);
}); ```

Answer 32 · 2019-08-13T16:13:03.000Z

Update to Chokidar v3, it should resolve this.