Why is Gatsby using watchers for `gatsby build`
jessepinho opened this issue · 10 comments
Summary
Relevant information
When running gatsby build, my CI system (Jenkins) often hits the system limit for file watchers. The limit on the system is 1024. I have only about 700 files in the repo, but this issue seems to occur when multiple builds are running simultaneously.
But my question is, why would gatsby build require file watchers? It seems that that would only be needed for watch commands. Is this a mistake?
Environment (if relevant)
System:
OS: macOS 10.14.5
CPU: (8) x64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
Shell: 3.2.57 - /bin/bash
Binaries:
Node: 10.15.3 - ~/.nvm/versions/node/v10.15.3/bin/node
Yarn: 1.5.1 - node_modules/.bin/yarn
npm: 6.4.1 - ~/.nvm/versions/node/v10.15.3/bin/npm
Languages:
Python: 2.7.16 - /usr/local/bin/python
Browsers:
Chrome: 75.0.3770.142
Firefox: 66.0.2
Safari: 12.1.1
npmPackages:
gatsby: ^2.13.14 => 2.13.14
gatsby-image: ^2.0.30 => 2.0.37
gatsby-plugin-react-helmet: ^3.0.7 => 3.0.12
gatsby-plugin-styled-components: ^3.0.6 => 3.0.7
gatsby-plugin-typescript: ^2.1.1 => 2.1.1
gatsby-transformer-remark: ^2.3.0 => 2.3.8
File contents (if changed)
Hm, these are all private as it's a private repo. Are these really needed?
gatsby-config.js: N/A
package.json: N/A
gatsby-node.js: N/A
gatsby-browser.js: N/A
gatsby-ssr.js: N/A
Because used package for globbing + watching (chokidar) doesn't have way to disable watching. We were looking for alternatives, but there were none that actually allow to disable watching. We could use different packages for builds (so only globbing, without watching) and develop (globbing + watching) - but this introduce potential to cause differences between builds and develop
Ah OK, makes sense. I should open an issue with chokidar then? I'll close this one in the meantime.
We could use different packages for builds (so only globbing, without watching) and develop (globbing + watching) - but this introduce potential to cause differences between builds and develop
There are differences between builds and develop anyway. I think it would be useful for issues like #11406 and others to use a different library for each case.
Hiya!
This issue has gone quiet. Spooky quiet. 👻
We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!
Thanks for being a part of the Gatsby community! 💪💜
Hey guys, I'm working on a project and seeing build failures due to this issue fairly often. Are there any plans to update gatsby to not use chokidar during build? If needed for globbing, maybe using the same library that chokidar uses for globbing?
Depends on what the package is and how you're going to solve it. Fs is brittle so it's not always easy to test all edge cases.
@jessepinho Great question! One I can think of right off the top of my head is hat choikdar seems to ignore changes when running in some environments (CI being the one in question). This made us skip some hot reloading tests on CircleCI 😒
I did attempt to move over to a different file watcher (watchpack) in #15854 (it fixed the CI issue)
Feel free to take that over if you like?
In the mean time, let's close this issue? We should move the discussion (of moving over to a different watcher in a separate issue in my opinion, feel free to open that!)
@jessepinho I was running into this issues as well in our restrictive Jenkins environment and settingCHOKIDAR_USEPOLLING=1 got around the file watcher limit problem.