High CPU usage

Question

High CPU usage

flynfish opened this issue 12 years ago · 38 comments

When running grunt watch, the process sits around 70% cpu usage on my mac (10.8.2). Is this a known issue?

Answer 1 · 2013-01-09T19:10:37.000Z

It's not. How many files are you watching and could you post your Gruntfile.js please? Does it happen with just watching or as tasks are ran? Also are you using grunt@0.4.0rc4 and grunt-contrib-watch@0.2.0a?

Answer 2 · 2013-01-09T19:26:09.000Z

Versions:
"grunt": "~0.3.17",
"grunt-contrib-watch": "~0.1.4",
"grunt-contrib": "~0.3.0"

As for the amount of files, since I am using bootstrap there are more 40 less files being watched. There really isn't much of a need for me to watch those though so perhaps that is one of the issues? Looks like it happens just when watching.

module.exports = function(grunt) {

  grunt.initConfig({

    // The lint task will run the build configuration and the application
    // JavaScript through JSHint and report any errors.  You can change the
    // options for this task, by reading this:
    // https://github.com/cowboy/grunt/blob/master/docs/task_lint.md
    lint: {
      files: [
        "public/**/*.js"
      ]
    },

    // The jst task compiles all application templates into JavaScript
    // functions with the underscore.js template function from 1.2.4.  You can
    // change the namespace and the template options, by reading this:
    // https://github.com/tbranyen/build-tasks/tree/master/jst
    //
    // The concat task depends on this file to exist, so if you decide to
    // remove this, ensure concat is updated accordingly.
    jst: {
      "public/js/templates.js": [
        "views/templates/**/*.html"
      ]
    },

    // The concatenate task is used here to merge the almond require/define
    // shim and the templates into the application code.  It's named
    // dist/debug/require.js, because we want to only load one script file in
    // index.html.
    concat: {
      "public/js/debug.js": [
        "public/js/vendor/*.js",
        "public/js/vendor/bootstrap/*.js",
        "public/js/templates.js",
        "public/js/app/*.js",
        "public/js/app/models/*.js",
        "public/js/app/views/*.js",
        "public/js/app/collections/*.js",
        "public/js/app/views/nests/*.js"
      ]
    },

    // This task uses the MinCSS Node.js project to take all your CSS files in
    // order and concatenate them into a single CSS file named index.css.  It
    // also minifies all the CSS as well.  This is named index.css, because we
    // only want to load one stylesheet in index.html.
    mincss: {
      "public/dist/release.css": [
        "public/stylesheets/*.css"
      ]
    },

    // Takes the built require.js file and minifies it for filesize benefits.
    min: {
      "public/dist/release.js": [
        "public/js/debug.js"
      ]
    },

    // less to CSS
    less: {
      compile: {
        files: {
          "public/stylesheets/style.css": "public/stylesheets/style.less",
          "public/stylesheets/nest.css": "public/stylesheets/nest.less",
          "public/stylesheets/style.css": "public/stylesheets/style.less",
          "public/stylesheets/vendor/bootstrap/bootstrap.css": "public/stylesheets/vendor/bootstrap/less/bootstrap.less",
          "public/stylesheets/vendor/fontawesome/font-awesome.css": "public/stylesheets/vendor/fontawesome/less/font-awesome.less"
        }
      }
    },

    watch: {
      scripts: {
        files: '**/*.less',
        tasks: ['less']
      },
      templates: {
        files: 'views/templates/**/*.html',
        tasks: ['jst']
      }
    }

  });

  // The default task will remove all contents inside the dist/ folder, lint
  // all your code, precompile all the underscore templates into
  // dist/debug/templates.js, compile all the application code into
  // dist/debug/require.js, and then concatenate the require/define shim
  // almond.js and dist/debug/templates.js into the require.js file.
  grunt.registerTask("default", "concat min less mincss jst");
  grunt.loadNpmTasks('grunt-contrib');
};

Answer 3 · 2013-01-09T19:34:05.000Z

Thanks for the info. If you get a chance, could you try the latest grunt and grunt-contrib-watch? Let me know if it still runs up the CPU. We've made a lot of changes since v0.1.4 in the watch. I'll try your setup shortly to see if I can duplicate. Thanks!

Answer 4 · 2013-01-09T20:41:46.000Z

I updated to the lastest grunt and grunt-contrib versions and now the process never gets above 1% cpu, so I would say you guys did some awesome work on that update!

When running watch now and updating a less file, I get this error:

running "less:compile" (less) task
Warning: Object #<Object> has no method 'expandFiles' Use --force to continue.

Aborted due to warnings.

Any ideas on what I need to change to get it to work? (I already update the registerTask default alias to use an array instead of string"

Answer 5 · 2013-01-09T20:52:41.000Z

Awesome, glad to hear!

To get less working, you'll need to install the Grunt v0.4 version with: npm install git://github.com/gruntjs/grunt-contrib-less.git. In fact all of your plugins will need updating if moving to Grunt v0.4 (https://github.com/gruntjs/grunt/wiki/Upgrading-from-0.3-to-0.4). FWIW, I use Grunt v0.4 exclusively. Just know it is still under development so not all grunt plugins support it yet and things can change.

Another thing to try if staying on Grunt v0.3 for the time being. Try uninstalling grunt-contrib-watch and using the previously built-in watch task in Grunt v0.3. grunt-contrib-watch is intended as a replacement for the watch task in Grunt v0.4. We just had Grunt v0.3 support on the v0.1.x branch to get some early testers. :)

Answer 6 · 2013-01-09T22:37:02.000Z

I tried upgrading to grunt .4 and upgrading the grunt plugins but I ran into some issues I don't have time to figure out. I removed the bootstrap less files from being watched and that dropped the cpu to under 5%.

Answer 7 · 2013-01-28T17:14:27.000Z

Hopefully the other issues you got figured out :) Otherwise please reopen if one of those is with the watch task, thanks!

Answer 8 · 2013-03-23T19:02:08.000Z

I'm having the same issue on my Windows 8, cpu level near 70%, watching just a few files.

Answer 9 · 2013-05-22T16:47:30.000Z

Have issue with high load too.

Versions:
"grunt": "~0.4.1",
"grunt-contrib-watch": "~0.4.3",
"grunt-contrib-sass": "~0.3.0"

System: Ubuntu 12.04 LTS

Gruntfile:

...
   watch: {
      sass: {
        files: ['app/stylesheets/{,**/}*.scss'],
        tasks: ['sass:dev']
      },
      html: {
        files: ['app/index.html'],
        tasks: ['html:dev']
      }
    },

  html: {
      dev: {
        options: {
          src: 'app/index.html',
          dest: 'temp/index.html',
          vars: config('development')
        }
      }
    },

    sass: {
      dev: {
        options: {
          style: 'expanded'
        },
        files: {
          'temp/stylesheets/application.css': 'app/stylesheets/application.scss'
        }
      }
    },
...

Answer 10 · 2013-05-22T16:58:30.000Z

Try running with --verbose to see which files it is watching. Also which version of node?

Answer 11 · 2013-05-22T17:35:35.000Z

Task is watching 45 scss files and version of node is v0.10.4.

Answer 12 · 2013-05-22T18:02:36.000Z

I'm curious if the problem is within this lib or just watching in general (as that greatly differs on environments). Are you game to try some examples for me?

Could you first make sure any previous node processes are not running with killall -9 node and then run this example to see if that causes the CPU spike?

var fs = require('fs');
[
  // Please add a bunch more files
  'app/stylesheets/application.scss',
].forEach(function(file) {
  fs.watch(file, function(event, filename) {
    console.log(file + ' was ' + event);
  });
});

Answer 13 · 2013-05-23T12:51:07.000Z

Not fully understand, how to run this example. Can you explain, please =)

Answer 14 · 2013-05-23T16:05:56.000Z

Oh that example will watch files using the core node.js api because I want to see if the problem is from this watch task or with any file watching.

Throw that example into a file named test.js. Add a bunch more actual file paths and then do this in your terminal: killall -9 node && node test.js to kill any other running node process and then run the example. Thanks!

Answer 15 · 2013-05-23T17:00:21.000Z

Thank you for explain) I can't find other process of node on machine and have this error on start:

vagrant@precise32:/vagrant$ killall -9 node                
node: no process found
vagrant@precise32:/vagrant$ node test.js

fs.js:1051
    throw errnoException(process._errno, 'watch');
          ^
Error: watch ENOENT
    at errnoException (fs.js:1019:11)
    at FSWatcher.start (fs.js:1051:11)
    at Object.fs.watch (fs.js:1076:11)
    at /vagrant/test.js:31:6
    at Array.forEach (native)
    at Object.<anonymous> (/vagrant/test.js:30:3)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)

Answer 16 · 2013-05-23T17:04:40.000Z

Thanks for giving it go. That error means one of the files you've tried doesn't exist. Could you try on only files that exists?

Answer 17 · 2013-05-24T08:17:16.000Z

Thanks, you're right) It's my bad knowledge of english =(
This is results:
configuration

watch: {
      sass: {
        files: ['app/stylesheets/application.scss',
                'app/stylesheets/app/{,**/}*.scss',
                'app/stylesheets/lib/{,**/}*.scss'
               ],
        tasks: ['sass:dev']
      },
sass: {
      dev: {
        options: {
          style: 'expanded'
        },
        files: {
          'temp/stylesheets/application.css': 'app/stylesheets/application.scss'
        }
      },

screenshot of htop when run watch:sass

configuration

var fs = require('fs');
[
  // Please add a bunch more files
  'app/stylesheets/application.scss',
  'app/stylesheets/app/_authentication.scss',
  'app/stylesheets/app/_dynamic-content.scss',
  'app/stylesheets/app/_main.scss',
  'app/stylesheets/app/_news-feed.scss',
  'app/stylesheets/app/crime-reporting/_incident-show.scss',
  'app/stylesheets/app/crime-reporting/_incidents.scss',
  'app/stylesheets/app/crime-reporting/_print-content.scss',
  'app/stylesheets/app/crime-reporting/_subjects.scss',
  'app/stylesheets/app/crime-reporting/_video-edit.scss',
  'app/stylesheets/app/features/_comments.scss',
  'app/stylesheets/app/features/_entities.scss',
  'app/stylesheets/app/features/_groups.scss',
  'app/stylesheets/app/features/_messages.scss',
  'app/stylesheets/app/features/_premises.scss',
  'app/stylesheets/app/features/_statistics.scss',
  'app/stylesheets/app/features/_users.scss',
  'app/stylesheets/app/headers/_action-bar.scss',
  'app/stylesheets/app/headers/_header.scss',
  'app/stylesheets/app/plugins/_custom-redactor.scss',
  'app/stylesheets/app/plugins/_on-off-switch.scss',
  'app/stylesheets/app/sidebars/_left.scss',
  'app/stylesheets/app/sidebars/_right.scss',
  'app/stylesheets/app/ui/_flashes.scss',
  'app/stylesheets/app/ui/_forms.scss',
  'app/stylesheets/app/ui/_tables.scss',
  'app/stylesheets/lib/_lt-ie7.scss',
  'app/stylesheets/lib/_lt-ie8.scss',
  'app/stylesheets/lib/_lt-ie9.scss',
  'app/stylesheets/lib/_mixins.scss',
  'app/stylesheets/lib/bootstrap/_custom-icons.scss',
  'app/stylesheets/lib/bootstrap/_ie7buttonfix.scss',
  'app/stylesheets/lib/bootstrap/_ie8buttonfix.scss',
  'app/stylesheets/lib/bootstrap/_mixins-custom.scss',
  'app/stylesheets/lib/bootstrap/_modal.scss',
  'app/stylesheets/lib/bootstrap/_overrides.scss',
  'app/stylesheets/lib/bootstrap/_popover.scss'
].forEach(function(file) {
  fs.watch(file, function(event, filename) {
    console.log(file + ' was ' + event);
  });
});

screenshot of htop when run node test.js

Answer 18 · 2013-05-24T16:26:30.000Z

Hmm bummer. I'll reopen although I'm not sure how to fix as I can't reproduce and I am out of ideas. I really appreciate you trying those examples for me @Jesterovskiy! If someone has this issue and discovers a fix please let us know! Thanks.

Answer 19 · 2013-05-24T22:01:38.000Z

Just chiming in to say that we're experiencing the same issue watching around 100 files. We ran the same test above and didn't see any serious CPU usage, so no idea what's causing it.

Answer 20 · 2013-05-27T08:05:53.000Z

Some test with https://github.com/shama/gaze shows same situation: 5 process and quiescent load 20-23% of cpu. I will dig deeper =)

Answer 21 · 2013-05-27T08:43:05.000Z

I found the problem! =) In Gaze by default interval sets https://github.com/shama/gaze/blob/master/lib/gaze.js#L36 but in fs lib we see different default value https://github.com/joyent/node/blob/master/lib/fs.js#L1128 I changed this value in my test file and the result - no more high cpu load =)
configuration

var fs = require('fs'),
    opts = { persistent: true, interval: 5007 };
[
  // Please add a bunch more files
  'app/stylesheets/application.scss',
  'app/stylesheets/app/_authentication.scss',
  'app/stylesheets/app/_dynamic-content.scss',
  'app/stylesheets/app/_main.scss',
  'app/stylesheets/app/_news-feed.scss',
  'app/stylesheets/app/crime-reporting/_incident-show.scss',
  'app/stylesheets/app/crime-reporting/_incidents.scss',
  'app/stylesheets/app/crime-reporting/_print-content.scss',
  'app/stylesheets/app/crime-reporting/_subjects.scss',
  'app/stylesheets/app/crime-reporting/_video-edit.scss',
  'app/stylesheets/app/features/_comments.scss',
  'app/stylesheets/app/features/_entities.scss',
  'app/stylesheets/app/features/_groups.scss',
  'app/stylesheets/app/features/_messages.scss',
  'app/stylesheets/app/features/_premises.scss',
  'app/stylesheets/app/features/_statistics.scss',
  'app/stylesheets/app/features/_users.scss',
  'app/stylesheets/app/headers/_action-bar.scss',
  'app/stylesheets/app/headers/_header.scss',
  'app/stylesheets/app/plugins/_custom-redactor.scss',
  'app/stylesheets/app/plugins/_on-off-switch.scss',
  'app/stylesheets/app/sidebars/_left.scss',
  'app/stylesheets/app/sidebars/_right.scss',
  'app/stylesheets/app/ui/_flashes.scss',
  'app/stylesheets/app/ui/_forms.scss',
  'app/stylesheets/app/ui/_tables.scss',
  'app/stylesheets/lib/_lt-ie7.scss',
  'app/stylesheets/lib/_lt-ie8.scss',
  'app/stylesheets/lib/_lt-ie9.scss',
  'app/stylesheets/lib/_mixins.scss',
  'app/stylesheets/lib/bootstrap/_custom-icons.scss',
  'app/stylesheets/lib/bootstrap/_ie7buttonfix.scss',
  'app/stylesheets/lib/bootstrap/_ie8buttonfix.scss',
  'app/stylesheets/lib/bootstrap/_mixins-custom.scss',
  'app/stylesheets/lib/bootstrap/_modal.scss',
  'app/stylesheets/lib/bootstrap/_overrides.scss',
  'app/stylesheets/lib/bootstrap/_popover.scss'
].forEach(function(file) {
  fs.watchFile(file, opts, function(event, filename) {
    console.log(file + ' was ' + event);
  });
});

Answer 22 · 2013-05-27T17:47:38.000Z

@Jesterovskiy Awesome find! Thanks for looking into this! That makes a lot of sense. I'll have to increase the default interval for polling in gaze. In the meantime that setting is available through the options: options: { interval: 5007 }. Thanks! :D

Answer 23 · 2014-04-21T23:24:09.000Z

I see that this has been closed but I still see grunt using about 9% (of a 2.4 i5) on a watch of a few dozen files on OS X. Doesn't seem like a lot but if I leave the watch running all day it keeps power management from kicking in and costs me battery life. I'll play around with that polling interval option, I just wanted to mention it here so that you know the defaults aren't ideal.

Answer 24 · 2014-04-22T03:10:48.000Z

FWIW, the next version of gaze uses native OS events for everything and will dramatically reduce the CPU usage. In the meantime, just make sure it's not matching unintended files grunt watch -v, set a higher interval.

Answer 25 · 2014-12-10T11:08:52.000Z

Hi,

I know its a very old issue but i face this problem again on My windows 8.1 PC.

Its regarding the Sails.js mvc. I have tried all the above suggestions but these were not working for me. I have to commented out the codes under Create watcher per target line from the file tasks/watch.js of this module.

I just stop the watchers to pushing items on it under a infinite type loop. I just did it so don't know what happened further.

Hope it will help others temporarily.

Thanks

Answer 26 · 2014-12-14T05:26:46.000Z

Also seeing a CPU usage on a Windows 7 machine. And high memory usage (> 100MB).

Answer 27 · 2014-12-20T15:15:37.000Z

Had the same problem.
It was caused by Grunt watching "node_modules" directory.
https://github.com/gruntjs/grunt-contrib-watch#why-is-the-watch-devouring-all-my-memorycpu

Answer 28 · 2015-02-06T14:40:25.000Z

i have the same issue!

Answer 29 · 2015-02-16T16:15:13.000Z

I am also experiencing this issue. It starts out running at around 10-20%, then after about five minutes it shoots up to 99% CPU and stays there until I stop the watch task. I was unaware this was happening on my desktop but when I run it on my laptop it gets very warm and the fan spins constantly, which is what alerted me to the issue.

Edit: Just noticed the comment about watching the node_modules folder, that was my problem as well. Once I specified the watch directories to exclude that it fixed the issue.

Answer 30 · 2015-02-16T16:42:57.000Z

@pwdonald exactly same for me. but i have node_modules excluded already...

Answer 31 · 2015-02-25T13:47:58.000Z

Have the same problem on Yosemite on mac mini, 50% CPU for couple hundreds of scripts. I use the same configuration on fedora and grunt uses less than 1%

Answer 32 · 2015-02-25T15:42:16.000Z

I've found out that updating gaze to 0.6.4 solves my issue. PR about it: #412

Answer 33 · 2015-06-16T15:29:35.000Z

@tikosar spot on! I was watching node_modules! tku

Answer 34 · 2015-09-16T22:52:19.000Z

I've had the issue of watching node_modules/ as well. I've added

   '!node_modules/**/*.*'

like:

    watch: {
        options: {
            livereload: true
        },
        express: {
            files: ['**/*.js', '!node_modules/**/*.*'],

Should you maybe disable watching node_modules/ by default since I don't see why anyone would do that anyways and they could always enable it manually?

Answer 35 · 2015-09-25T06:20:17.000Z

I'm seeing the same disparity in CPU usage between Linux (lower/more manageable) and Mac (high). I'm using 0.6.1.

Answer 36 · 2016-03-08T16:24:24.000Z

obviously ymmv, but just thought I mention another potential biggie that caused slowdown on my Mac which was solved by ignoring the bower_components folder, e.g. !app/bower_components/**/*.*

before : ~3100 files using ~%60 CPU usage
after: ~10 files using ~5% CPU usage

versions:

grunt-contrib-watch v0.6.1
grunt-cli v0.1.13
grunt v0.4.5
node v5.5.0

I would have thought watching 3100 files would not be such a problem giving the Mac's FSEvent API is supposed to be fast and light weight.

But, I've just noticed that the grunt-contrib-watch package.json doesn't explicitly mention the node module https://www.npmjs.com/package/fsevents as a dependancy and uses gaze which at first glance also doest use node's fsevent. (this was just a cursory check, but given the behaviour it would seem it doesn't)

I guess a potential fix (for very large projects) is to use fsevent on Mac (either directly in grunt-watch or by patching node's gaze)

Answer 37 · 2016-03-10T10:49:07.000Z

My special dumb case:

Took me 3h (of looking in watch source and gaze and resetting Gruntfile.js to a minimum and update packages one by one and changing versions, etc) to realize that my task was configured as follows:

grunt.registerTask('watch', ['watch']);

Guess what? Recursion! CPU went in 100% instantly, no info in terminal, verbose mode showed watch task running continuously.

That was stupid but made me laugh in the end.

I don't think my solution will help someone but I posted it just to cover all scenarios...

Answer 38 · 2017-06-30T17:39:40.000Z

Someone had composer'd a /vendor/ dir in my theme. Ignoring that with '!/vendor/' did the trick for me. Thanks!