guigrpa/oao

Bootstrap is flaky with yarn 0.25.4 onward

eliperelman opened this issue · 14 comments

eliperelman commented

Ever since yarn upgraded past 0.25.4, I am unable to run bootstrap in Neutrino, there seems to be problems linking packages. I tried switching to workspaces as well but couldn't because of compatibility issues with yarn.

Example log:

https://travis-ci.org/mozilla-neutrino/neutrino-dev/jobs/280543596

Have you noticed anything put of sorts with yarn that would prevent us from upgrading? Right now I have to pin yarn to keep things from busting.

guigrpa commented

Hi Eli, thanks for the report.

I'll have a look into the linking problems you describe. The logs only say Killed, so I wonder what the issue can be. Any Neutrino branch I can use to reproduce the problem?

Regarding the avalanche of warnings coming from nyc, I'd also noticed them on several of my packages, even without oao.

Finally, what compatibility issues are you referring to regarding workspaces? Neutrino - yarn? Oao - yarn? I'm also finding issues migrating Storyboard to workspaces, btw.

eliperelman commented

Any Neutrino branch I can use to reproduce the problem?

I can reproduce this problem consistently locally on master.

Finally, what compatibility issues are you referring to regarding workspaces? Neutrino - yarn? Oao - yarn?

It's hard to remember at the moment, but I remember not even being able to upgrade or install packages properly, but maybe that was yarn and not oao. Glad to see it's not just me then. Let me know if I can provide anything in tracking this down, or if we have to chock it up to yarn issues.

guigrpa commented

No problem on my side with yarn v1.0.2, OS X v10.12, Node v8.5.0 and npm v5.3.0…

guigrpa commented

Wait, I hadn't cleaned all submodule node_modules directories.

I've checked again with yarn v1.1.0 (all other versions same as before) after npx oao clean and:

  • Noticed fewer warnings as before
  • The whole process was faster (by 30s)
  • I still had no problems

I wonder what could be the problem in your case. Maybe Node v6 in your case vs. v8 in mine?

guigrpa commented

Switched to Node v6.11.3, npm v3.10.10, yarn v1.1.0 and again no problems.

edmorley commented

I can repro with node 8.6.0, yarn 1.1.0, MSYS2 on Windows 10 x64.

STR:

  1. Clone https://github.com/mozilla-neutrino/neutrino-dev revision 484c9882eeec
  2. If not a clean checkout, do: rm -rf node_modules packages/*/node_modules
  3. yarn && yarn bootstrap (the latter being an alias for oao bootstrap)
  4. yarn deps:outdated (an alias for oao outdated)

Expected:
No errors (aside from the usual case-insensitive file noise).

Actual:
This error: https://emorley.pastebin.mozilla.org/9069041

edmorley commented

I'm presuming there are really two bugs here: (1) the error from yarn not being captured and output to the console (there isn't a yarn debug error log file created either), (2) some other bug that the stderr from yarn would help identify.

guigrpa commented

@edmorley I could reproduce your error with oao outdated on OS X with yarn 1.1.0. I traced it to the following lines in /src/utils/shell:

    const child = execa.shell(cmd, {
      cwd: cwd || '.',
      // Workaround for Node.js bug: https://github.com/nodejs/node/issues/10836
      // See also: https://github.com/yarnpkg/yarn/issues/2462
      stdio:
        process.platform === 'win32' ? ['ignore', 'pipe', 'pipe'] : undefined,
    });
    // ...
    const { code, stdout, stderr } = await child;

If the command is something simple (say, ls -al, or even yarn outdated with previous yarn versions, e.g. 0.25.1), execa.shell works correctly. With v1.1.0 of yarn, though, it seems to break. Any idea why?

On the other hand, do you believe this is the same issue as the OP?

edmorley commented

Hmm I think I've found the issue.

As of yarn 0.26.1, yarn outdated now intentionally returns exit code 1 if there were outdated dependencies (yarnpkg/yarn#3483). It is this that is making the command look as though it had failed. Oao will need to support the exit-code being non-zero in this case.

Another issue, is that when running yarn deps:outdated (which just runs oao outdated), then there is less output to the console then when running oao outdated directly (the stdout that shows the outdated dependencies doesn't appear). That is presumably a yarn bug - I'll see if I can find the regression range and file a yarn issue.

edmorley commented

For the "less output" part, I've tracked it down to another intentional change - yarnpkg/yarn#3536, which can be worked around by setting YARN_SILENT to 0 in oao. However I've also asked if the yarn change could be reverted - yarnpkg/yarn#4615.

edmorley commented

To summarise: for newer yarn support, oao needs to (a) expect yarn outdated to return non-zero if there were outdated packages found, (b) set process.env.YARN_SILENT = 0 before running any yarn child processes (both in the oao outdated command and elsewhere).

guigrpa commented

@edmorley I've just released oao v0.10.2 with both changes you suggest:

  • Ignore the non-zero error code returned by yarn outdated
  • Set YARN_SILENT = 0

Now yarn deps:outdated should work correctly in neutrino-dev, could you please check?

@eliperelman Could you please check whether oao v0.10.2 provides more details on your original yarn bootstrap error? Thanks.

  • 👍2
edmorley commented

Now yarn deps:outdated should work correctly in neutrino-dev, could you please check?

Works great now, thank you :-)

guigrpa commented

Can probably be closed now, given the discussion in #51.