yarn_install tries to remove bazel binary on windows, fails

[filipesilva]

🐞 bug report

Affected Rule

The issue is caused by the rule: yarn_install

Is this a regression?

Yes, the previous version in which this bug was not present was: 0.31.1 I think.

Description

When installing on node modules on windows, we see "EPERM: operation not permitted, unlink 'C:\\users\\circleci\\ng\\node_modules\\@bazel\\bazel-win32_x64\\bazel-0.27.0-windows-x86_64.exe'" errors in https://circleci.com/gh/angular/angular/380183.

I think this only started happening after angular/angular#31325

🔬 Minimal Reproduction

I saw it on angular/angular#31266. But it doesn't seem related to the PR proper. It looks like yarn install --frozen-lockfile --non-interactive && yarn bazel info is enough to reproduce it on https://github.com/angular/angular master as of today.

🔥 Exception or Error

ERROR: An error occurred during the fetch of repository 'npm':
   yarn_install failed: $ node tools/yarn/check-yarn.js
...
[4/5] Linking dependencies...
info If you think this is a bug, please open a bug report with the information provided in "C:\\users\\circleci\\ng\\yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
 (warning " > tsickle@0.34.3" has incorrect peer dependency "typescript@~3.3.1".
error An unexpected error occurred: "EPERM: operation not permitted, unlink 'C:\\users\\circleci\\ng\\node_modules\\@bazel\\bazel-win32_x64\\bazel-0.27.0-windows-x86_64.exe'".
Process stalled
Active handles:
  - Socket
  - Socket
  - Socket
)

🌍 Your Environment

Operating System:

  
Windows 10
  

Output of bazel version:

  
0.27.0
  

Rules version (SHA):

  
0.32.2
  

Anything else relevant?

cc @gregmagolan @alexeagle

[gregmagolan]

I did discuss with @alexeagle a while ago about changing @bazel/bazel to download the bazel binary on demand when run similar to what webdriver & puppeteer do. It could put the binary in a /tmp folder and create put a .json file under node_modules that points to it. That way the binary is not run out of node_modules. The downside there is that the binary won't be put in the yarn cache or npm cache which means it needs to be downloaded more often.

[gregmagolan]

The bazel process is spawned by @bazel/bazel index.js like so

if (require.main === module) {
  /** Starts a new synchronous child process that runs Bazel with the specified arguments. */
  const bazelProcess = spawnSync(getNativeBinary(), process.argv.slice(2), {stdio: 'inherit'});

  // Ensure that this wrapper script exits with the same exit code as the child process.
  process.exit(bazelProcess.status);
}

We could still use the native @bazel npm packages so the yarn & npm caches are used but then copy the binary into /tmp, drop a .json file in node_modules to point to the /tmp executable so we don't have to copy each time, and then run the Bazel out of /tmp.

[filipesilva]

I've a workaround for now: installing @bazel/bazel globally and using that for an initial bazel info. Then use yarn bazel for subsequent runs. That seems to get around the problem.

Although I'm not sure why, I guess the initial bazel info installed necessary stuff? But why are existing node_modules files deleted in the first place?

[meteorcloudy]

Yes, bazel info will try to load the external repository, so node_modules could be installed at that time.

But I couldn't reproduce the failure with

yarn install --frozen-lockfile --non-interactive && yarn bazel info

Do I need to do something else?

[filipesilva]

@meteorcloudy I'm not sure what else is relevant. I see it consistently on angular/angular#31266 (comment).

[meteorcloudy]

Ha, I can reproduce by changing the Bazel version in package.json then running yarn bazel info

I think the reason is yarn bazel info triggers yarn_install rule to reinstall Bazel, but the bazel binary is currently opened, so it cannot delete the existing Bazel binary.

But what I don't understand is, why this is happening on CI. How did yarn bazel info triggered a rerun of yarn_install after yarn install --frozen-lockfile --non-interactive step.?

[meteorcloudy]

So basically, we want yarn to release the file hand of the bazel binary after it spawned bazel info command. Not sure if it's doable.

[filipesilva]

I don't think that's doable because it's not a yarn thing, it's a Windows thing. As long as we're using the Bazel executable in the node_modules, we can't delete it.

[meteorcloudy]

You can actually delete the bazel binary while it's running, but not when the binary is opened in another process. But generally, this may not be a good idea. I think we should just avoid upgrading Bazel in a repository rule.

But I still couldn't reproduce the failure with a clean Angular repo, do you know what could cause the yarn_install rule to reinstall Bazel? There's only a copy .circleci\bazel_win.rc $env:ProgramData\bazel.bazelrc step in between, but looks harmless.

[filipesilva]

@gregmagolan might know what would cause yarn_install to reinstall the modules. Perhaps a different version of node/yarn?

[glebmachine]

Hi! Any updates, still got this error
I't looks like yarn_install totaly ignores currently installed node_modules

[glebmachine]

Why build task should install deps?

[SiddAjmera]

I'm getting the same issue running Bazel 2.0.0

[haoolii]

Bazel 3.3.0 still has the same issue.

[hiepxanh]

1. install everything with npm install then run ng serve
   failed to run on Windows with powershell as administrator, already try with yarn and npm

2. remove node_modules
   try again with with powershell as administrator, then yarn and ng serve
   
   run successful

so, we need yearn, and run as administrator to trigger window defend firewall, or we will get operation not permitted, unlink

TESTED with window 10, AMD system, latest yarn and angular bazel examples from rules_nodejs

[alexeagle]

I think the workaround is to install Bazel yourself (npm i -g @bazel/bazelisk would be a good way) so that the Bazel binary is not one managed within the project. Could you check if that solves it?

[hiepxanh]

@alexeagle confirm it work sir. finally, I can run bazel smoothly on windows @filipesilva can you try it?
@alexeagle bazel is really fast, I'm so exciting, it crazy fast, is that it using all my cpu core ? I have ryzen AMD 8 core and I'm so happy with this speed

[alexeagle]

Yes Bazel parallelizes the build by breaking into many subprocesses which use all available cores. If you add remote execution you can add unlimited CPU. It depends on having a wide dependency graph where many steps can be run independently (test targets/shards is a good example)
Thanks for verifying the workaround.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs in two weeks. Collaborators can add a "cleanup" or "need: discussion" label to keep it open indefinitely. Thanks for your contributions to rules_nodejs!

[github-actions]

This issue was automatically closed because it went two weeks without a reply since it was labeled "Can Close?"

[basherr]

@hiepxanh @alexeagle What are the steps to fix it? I am still getting similar error.