CLI functionality related to aggregator preventing yarn commands 1.19-1.22 - Reverted to 1.16


^ Problems started with upgrading to 1.19 and then snowballed into 5 hours of hell trying to figure out what went wrong.

Do you want to request a feature or report a bug?
error An unexpected error occurred: "expected workspace package to exist for \"rimraf\""
It started with some dev dependency in a workspace repo and after messing around with nohoist settings I started noticing that the same modules keep coming up. Rimraf, inquirer, react-dev-tools.. generally things that were listed many times over in transitive dependencies. Read through many different similar issues regarding symlinks but ended up being unable to add new modules through cli which was the problem I was trying to address for automation.

What is the current behavior?

These commands do not seem to work in version v1.19.2 through v1.22.0-20191201.1637:

yarn workspace @scope/A add uuid lodash.get --dev
cd packages/A && yarn add uuid lodash.get --dev
yarn workspace @scope/A add js-cookie@^2.2.1 -D
cd packages/A && yarn add lodash.get@^2.2.1 --dev

Tried yarn cache clear, removing node_modules, reinstalling yarn to no avail.

If the current behavior is a bug, please provide the steps to reproduce.

Sorry, work repos.

What is the expected behavior?
Workspaces command and yarn add should work from cli.

Please mention your node.js, yarn and operating system version.
Tested on the following:

Node:
v8.9.4
v8.15.1
v10.11.0
v10.16.0
v10.16.2

Yarn:
v1.19.2
v1.21.0
v1.22.0-20191201.1637

OS:
MacOS Mojave 10.14.6

For anyone that comes looking for a quick fix.. I was able to get back to a working state by installing v16.0 https://raw.githubusercontent.com/karlhorky/homebrew-core/f8bc263f93dcfd1cc50c98976a8503c8a7cb032f/Formula/yarn.rb via homebrew. I'm gonna try seeing if the update with 1.17.4 works and go as far up as I can without losing functionality.

~~I was able to go up to 1.17.4 but 1.19 and up the same issues come up.~~

1.17.4 continues to have lots of symlink issues which were addressed in 1.19 but is not usable currently for me. 1.16 seems more stable but is not secure... sigh. Please let me know how I can help. Need a break and will check in later.

A clue about symlink issues while running --verbose. The uuid module keeps missing even as far back as 1.16. The common pattern right now appears to be modules that are used in many other modules but are listed as a peerDependency in a workspace can run into issues with workspace aggregator during certain cli commands. This could be because a lot of commonly used modules are listed as peerDependencies vs dependencies, following advice from https://yarnpkg.com/blog/2018/04/18/dependencies-done-right/.

From #6953
I'm wondering if 2.0 will address this with the above.. but still unsure why this became a bigger problem past 1.16. Is it because changes to 1.16 are being made with the same lens applied for code going in for 2.0? If so, should we just consider pinning our version of yarn and wait for 2.0 if we are using workspaces?

I think we're running into this issue at @thumbtack. In particular, this affects us when trying to upgrade a package within a workspace.

This snippet should reproduce the issue:

git clone git@github.com:thumbtack/thumbprint.git && cd thumbprint && yarn && cd packages/thumbprint-react && yarn upgrade @thumbtack/thumbprint-tokens --latest

This is the error you'll see:

error An unexpected error occurred: "expected workspace package to exist for \"@babel/plugin-transform-dotall-regex\"".
info If you think this is a bug, please open a bug report with the information provided in "/Users/<redacted>/thumbprint/packages/thumbprint-react/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/upgrade for documentation about this command.

Confirmed that downgrading Yarn to 1.16.0 works. Thanks for digging into this, @DavideDaniel. Saved me lots of time!

Here are downgrade instructions for those that installed with Homebrew: thumbtack/thumbprint#592 (comment)

Downgrading worked for me. But I'd definitely like to see this fixed in the latest version of yarn so I'm not lagging behind by 5 minor versions... yikes!

Downgrading to v1.19.0 worked for me. Looking forward to fix this~

Downgrading to v1.19.0 worked for me. Looking forward to fix this~

Same here.

Any update on this?

In my lerna monorepo I have some packages (like husky and lint-staged) installed at the root

(My .yarnrc file has --ignore-workspace-root-check true)

When I try to upgrade packages (using yarn upgrade-interactive --latest) I always get Invariant Violation: expected workspace package to exist for "" where the quoted portion includes the name of the root package. Downgrading to yarn 1.17.0 has no such issue

I just want to hop in and clarify my situation, in case it helps anyone else.

Failing Command

The commands that were failing for me were of the yarn workspace ${workspace} add ${package} variety.

Working Yarn Version

The indicated command is FAILING for me when using yarn=1.19.2-1 is SUCCEEDING for me when using yarn=1.19.1-1.

Install Older Yarn

First, follow the directions here: https://yarnpkg.com/lang/en/docs/install/#debian-stable

However, when it comes to the installation command, sudo apt update && sudo apt install -y --no-install-recommends yarn, you'll want to change yarn to yarn=${version}. You can find appropriate versions by doing apt-cache policy yarn.

Once you install a yarn version, you may also want to consider doing a sudo apt-mark hold yarn until this is resolved in later versions, otherwise apt upgrade will update your yarn again. You can confirm it is held by doing apt-mark showhold.

We are experiencing the same issue since updating to 1.21.1. Our message is as follows:

An unexpected error occurred: "expected workspace package to exist for \"@babel/traverse\"".

This is in a monorepo. When we downgrade to 1.19.0, the problem goes away. We don't want to do that, however, because of this recent security bulletin:

https://blog.npmjs.org/post/189618601100/binary-planting-with-the-npm-cli

Any suggestions?

Also experiencing this, here's our info – happy to provide anything else that might be useful.

Basic workspace monorepo, not using lerna.

Yarn version: 1.21.1

In:
yarn add ts-jest -D

Out:
error An unexpected error occurred: "expected workspace package to exist for \"string-width\"".

string-width is there in my node_modules. I've tried with both string-width in the main workspace node_modules and nohoisted.

When I downgrade to 1.16.0 I can complete the install.

My team is also experiencing this issue with version 1.21.1. However, as some other folks noted, all install commands work on prior versions (1.19.1 in our case)

This is blocking us from addressing the security issue documented here:

https://snyk.io/blog/understanding-filesystem-takeover-vulnerabilities-in-npm-javascript-package-manager/

The same issue for our team.

yarn 1.21.1
node v10.15.1

Invariant Violation: expected workspace package to exist for "core-js-compat"
yarn 1.21.1
node v13.2.0

Same issue with yarn 1.21.1 and node v12.14.1.

Same here.

yarn --version
1.21.1

node --version
v12.9.0

same here

node -v && yarn -v
v10.15.3
1.21.1

For those looking for a solution that doesn't involve changing the version of yarn you're using, editing package.json to include the version of the package you want and then running the yarn command again works just fine.

I was having a similar issue when running yarn workspace <workspace> add <pkg>, and resorted to manually amending that package's package.json and running yarn. Idea from this post.

@aryzing thanks for that suggestion and sharing a link to that post. However, in our team with use the --frozen-lockfile flag by default (by adding it to our .yarnrc file) to avoid accidental changes to dependencies. It seems like we would need to disable that feature in order to manually edit package.json files -- and as a result, risk accidental changes to the dependencies across all workspaces, unless I'm missing something.

Here is a stack trace (node v12.14.0 , yarn v1.21.1):

verbose 2.685597913 Invariant Violation: expected workspace package to exist for "eslint-plugin-react"
    at invariant (/usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:2314:15)
    at _loop2 (/usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:94898:9)
    at PackageHoister.init (/usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:94957:19)
    at PackageLinker.getFlatHoistedTree (/usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:48743:20)
    at PackageLinker.<anonymous> (/usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:48754:27)
    at Generator.next (<anonymous>)
    at step (/usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:310:30)
    at /usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:328:14
    at new Promise (<anonymous>)
    at new F (/usr/local/Cellar/yarn/1.21.1/libexec/lib/cli.js:5301:28)
error An unexpected error occurred: "expected workspace package to exist for \"eslint-plugin-react\"".

This appears to be a bug in the PackageHoister. See:

yarn/src/package-hoister.js

Lines 855 to 872 in c43f66d

    
           // Check if the destination is pointing to a sub folder of the virtualManifestName 
        
           // e.g. _project_/node_modules/workspace-aggregator-123456/node_modules/workspaceChild/node_modules/dependency 
        
           // This probably happened because the hoister was not able to hoist the workspace child to the root 
        
           // So we have to change the folder to the workspace package location 
        
           if (this.workspaceLayout && isWorkspaceEntry) { 
        
             const wspPkg = this.workspaceLayout.workspaces[keyParts[1]]; 
        
             invariant(wspPkg, `expected workspace package to exist for "${keyParts[1]}"`); 
        
             parts.splice(0, 4, wspPkg.loc); 
        
           } else { 
        
             if (this.config.modulesFolder) { 
        
               // remove the first part which will be the folder name and replace it with a 
        
               // hardcoded modules folder 
        
               parts.splice(0, 1, this.config.modulesFolder); 
        
             } else { 
        
               // first part will be the registry-specific module folder 
        
               parts.splice(0, 0, this.config.lockfileFolder); 
        
             } 
        
           }

This is a serious issue as it effectively prevents any package from being installed.

Any update on when this is supposed to be fixed?

This is a serious issue as it effectively prevents any package from being installed.

Any update on when this is supposed to be fixed?

you can manually adjust the package.json and run yarn install.
The only sad part about this is that you have to sort the packages yourself from A-Z when adding new packages 😂

Any update from yarn maintainers on whether this issue is even seen or recognized?

Just updated to 1.22.0 via Homebrew and ran into this issue as well...

Just updated to 1.22.0 via Homebrew and ran into this issue as well...

sorry, to me the version 1.22.0 don't solve the problem. 😞

I've run into this problem as well a couple of weeks ago. Monorepo using Lerna and yarn workspaces. All of a sudden adding packages causes An unexpected error occurred: "expected workspace package to exist for \"@typescript-eslint/eslint-plugin\"". Upgraded to 1.22.0, problem still persist.

Right now we have to manually add packages to the package.json and run yarn install.

Upgraded from 1.19.x to 1.22.0 and ran into this issue. yarn cache clean solved the issue for me.

I just tried again to upgrade from 1.16.x. This time to 1.22.0. The problem still exists for me. yarn cache clean did not help.

I get the same error when yarn adding something to a package of a monorepo:

error An unexpected error occurred: "expected workspace package to exist for "@storybook/react"".

@storybook/react is installed in the monorepo root's package.json in the devDependencies. When I change the monorepo root's devDependencies to dependencies, the issue disappears. I don't think you should do that but maybe it helps understanding and fixing the problem.

For me, the problem is a regression 1.19.1->1.19.2 (as reported here). Downgrading to 1.19.1 helped.

Building off of #7734 (comment)

In my case, yarn erroneously thinks that a dependency is actually a child package within my monorepo workspace. In other words, I author many packages in my monorepo that all have some mix of dependencies, devDependencies, or peerDependencies on slightly different versions of ts-node. This causes yarn to incorrectly classify ts-node as being one of the monorepo's own packages. It thinks I have a ./packages/ts-node subdirectory in my monorepo, but I don't.

I think yarn is realizing it cannot hoist ts-node because there are different versions in play, so it keeps them nested within packages/*/node_modules (good) but this causes it to believe that ts-node is owned by my monorepo. (bad)

Still failing for me:

The same issue for me with yarn v1.22.4

Fails for me too on 1.22.4. Manually adding the module to package.json did work around the issue, though.

If the Yarn team have just abandoned addressing v1.x.x issues, then a deprecation notice would be nice...

Still facing this issue.. downgrading to 1.19.1 fixed it

sudo apt remove yarn -y
sudo npm install yarn@1.19.1 -g

It is also possible to invoke a specific version of Yarn directly using npx:

npx yarn@1.19.1 upgrade-interactive

That way, is not necessary to downgrade the globally installed version. When using workspaces, it seems the last version that works is 1.19.1 (getting an error with 1.19.2).

To downgrade to the last known working version in homebrew, run:

brew unlink yarn
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/aacaa70e8931b2b005cb8f70703b48604f4a8d2a/Formula/yarn.rb
brew switch yarn 1.19.1

Te solution provided by @friederbluemle here is the best one IMHO

Is this ever going to be fixed? I am a bit concerned about the quality of yarn in this context.

I've updated to 1.22 from older version and encountered this issue in couple of projects. But it was resolved by analysis with yarn why .... Sometimes yarn creates strange hoisting configurations when packages take in their inner node_modules a package of older versions while they could use hoisted. It is usually resolved by removing packages that hold "wrong" versions of dependencies and reinstalling them.

That moment when you google a frustrating issue, find a promising thread on github.com... only to realize deep into the comments that.. you're the original author 🤦 sigh..

I got a new work computer.. same repos, most specs are the same, OS is Catalina this time.
We've been trying to pave a way forward to get to yarn 2 but it is no small feat so the fact that I ran into this again just shows that this probably deserves a patch for 1.x maintenance branch. In the meantime we are going to continue running yarn commands with @friederbluemle's suggested approach of using npx to pin the version to one where it works.

Is this ever going to be fixed? I am a bit concerned about the quality of yarn in this context.

I have a feeling effort has moved onto yarn 2 and I think someone else said it, yarn 1.x needs a deprecation notice if there is no appetite for maintaining 1.x

Is this ever going to be fixed? I am a bit concerned about the quality of yarn in this context.

I have a feeling effort has moved onto yarn 2 and I think someone else said it, yarn 1.x needs a deprecation notice if there is no appetite for maintaining 1.x

Is this issue actually present in Yarn 2.0? If not, I'll be switching over. I'm forever stuck with an out of date yarn version atm

I had success so far moving many devDependencies up to the root/solution project only and then rerunning yarn upgrade-interactive with a minimal selection of upgrades.

Side Note: I noticed along the way that sometimes it took two sequential runs of yarn (yarn install) for it to settle down and report success Already up-to-date.

Thanks to all the contributors to this thread, and to @friederbluemle for the npx solution. An alternative is to issue the command yarn policies set-version 1.19.1 from inside your project. This will let you to then use the standard yarn command and have it reference that desired version only when inside your project.

It is also possible to invoke a specific version of Yarn directly using npx:
npx yarn@1.19.1 upgrade-interactive

I'm experiencing an issue with difference between yarn.locks generated with yarn v1.19.1 and latest.

You could also use lerna to add a package without downgrading the yarn version:

npx lerna add <package>[@version] --scope=<workspace>

The command above will add the package to the specified workspace package.json and update yarn.lock.

will this ever be actually fixed?

Looks like no. They really want us to transition to yarn2

Same issue when using chalk >=3 in a Yarn workspace. I found 2 ways to fix this:

Moving chalk from devDependencies to dependencies. However this doesn't look good to me, so going for option 2
Downgrading chalk to v2.0.0

Downgrading to v1.19.0 worked for me. Looking forward to fix this~

Same here... downgrading to v1.19.0 really solved the problem running yarn set version 1.19.0 at root of the project.

node -v && yarn -v                                                  
v10.19.0
1.22.4

As a warning to others, I've also discovered that yarn add (and friends) will produce an incorrect node_modules hierarchy under the conditions that trigger this bug.

Notably, that issue also appears to affect 1.19.1 and below. Running a subsequent yarn install fixes the issue.

The same issue for me with yarn v1.22.4 and node v14.15.4

If you're one of the many who are impacted by this issue, don't get your hopes up for a bugfix – Yarn 1 appears to have been quietly deprecated, and the maintainers no longer appear to be reviewing pull-requests in this repository.

If Yarn 2 doesn't work for you, it's probably time to start looking at npm or pnpm.

If Yarn 2 doesn't work for you [...]

Open an issue with a reproduction so it can be fixed

@merceyz I've had a pull-request open for the past 146 days that fixes this issue. It has not been reviewed or commented upon by a single member of the Yarn team.

Additionally, on Jan 1, the GitHub issue template for this repository was quietly updated to mention that non-security bugfixes are no longer being considered.

Please do not encourage members of the community to waste their time creating reproduction-scenarios or patches for issues that the team has directly stated they have no intention of fixing.

Please read my comment again, I specifically quoted the part of your comment about Yarn 2

This issue was resolved for me by using:

yarn lerna add --scope [scope] [...]

Instead of:

yarn add [...]

This issue was resolved for me by using:
yarn lerna add --scope [scope] [...]

how you use it when you want to install more that one package ?

lerna add --scope="@client" @types/react @types/react-dom --dev

not works for me.

Same issue when using chalk >=3 in a Yarn workspace. I found 2 ways to fix this:

Moving chalk from devDependencies to dependencies. However this doesn't look to me, so going for option 2

Downgrading chalk to v2.0.0

@andreasonny83 thanks.
We have the same issue but with yargs (yeilding the error about cliui). Moving yargs to dependencies instead of devDependencies worked. But do you know why it works?

It stopped failing for me once I fixed my workspace names.

package.json
packages/
    hardhat/
        package.json
     ui/
        package.json

root package.json:

  "name": "web3-boilerplate",
  "private": true,
  "workspaces": ["packages/ui", "packages/hardhat"],
   //etc

packages/hardhat/package.json:

    "name": "@web3-boilerplate/hardhat",
    "version": "1.0.0",
    //etc

I previously had named the packages just hardhat and ui and this seems to have been causing the issue. Changing to @web3-boilerplate/hardhat and @web3-boilerplate/ui seems to have fixed it!!

@elenamik Your comment helps me a lot.

After I changed my root package.json name as same as sub-package's @{name}, "expected workspace package to exist for @babel/preset-env" Error disappeared.

Also, I've experienced unexpected package version unsync error.

I have installed jest@^26.x in root workspace, and my react-scripts@5 has nested dependency of jest@^27.x. Because of it, I got "expected workspace package to exist for jest" error message. (I've found the jest package dependency relationship by inspecting yarn.lock file).

So I upgrade root workspace's jest version with equal to react-scripts@5 (jest@^27.x)

After that, "expected workspace package to exist for jest" disappeared.

	// Check if the destination is pointing to a sub folder of the virtualManifestName
	// e.g. _project_/node_modules/workspace-aggregator-123456/node_modules/workspaceChild/node_modules/dependency
	// This probably happened because the hoister was not able to hoist the workspace child to the root
	// So we have to change the folder to the workspace package location
	if (this.workspaceLayout && isWorkspaceEntry) {
	const wspPkg = this.workspaceLayout.workspaces[keyParts[1]];
	invariant(wspPkg, `expected workspace package to exist for "${keyParts[1]}"`);
	parts.splice(0, 4, wspPkg.loc);
	} else {
	if (this.config.modulesFolder) {
	// remove the first part which will be the folder name and replace it with a
	// hardcoded modules folder
	parts.splice(0, 1, this.config.modulesFolder);
	} else {
	// first part will be the registry-specific module folder
	parts.splice(0, 0, this.config.lockfileFolder);
	}
	}