Yarn Berry: `yarn bin` does not yield a path

I recently tried switching from Yarn Classic to Yarn Berry (i.e., version 3 or 4) with Shakapacker, but failed. After debugging this issue further, I've identified a potential issue with the current implementation in the Shakapacker webpack(-dev-server) when the package.json contains a webpack script.

Expected behavior:

Using the latest version of Yarn Berry (i.e., version 4.0.0 currently), I am able to run the Shakapacker webpack command through yarn run webpack. Especially, I do not expect to create an infinite recursion loop.

Actual behavior:

When running yarn run webpack, I don't see any output. Instead, I can monitor that an infinite number of processes get started until my machine freezes or I cancel the execution.

Small, reproducible repo:

https://github.com/MrSerth/shakapacker-yarn-4

The repository contains four commits:

Adding a new (empty) Rails app, MrSerth/shakapacker-yarn-4@dd5a67b15825b8d7e832b3b38e35b00498545d4e
Adding Shakapacker and installing it according to the Readme, MrSerth/shakapacker-yarn-4@c27879ddcd58284c15ee24f554e5dd5bb33cb1ef
⚡ Allowing developers to use yarn run webpack for interacting with shakapacker, MrSerth/shakapacker-yarn-4@9fd56127af0dea13edac6d59418992f082b6313c. This is designed to provide an additional support for those developers regularly working with Webpack directly (without using Shakapacker). This mechanism worked since the beginning of Webpacker and is still working today when using Yarn Classic.
- ✅ Executing yarn run webpack works fine
- ✅ Executing yarn bin yields a single path, see Yarn Classic docs for yarn bin
Switching to Yarn Berry, MrSerth/shakapacker-yarn-4@7adaaab927501714ba0c51c982ba7f1adaa6d0c6. This breaks the app and workflow.
- ❌ Executing yarn run webpack never terminates, but instead invokes an infinite recursion.
- ❌ Executing yarn bin does not list a single path, but list all available binaries. See Yarn Berry docs for yarn bin.

The changed behaviour of yarn bin is problematic, since Shakapacker relies on the previous behaviour of yarn bin:

shakapacker/lib/shakapacker/runner.rb

Line 17 in a99167f

    
           @node_modules_bin_path = ENV["SHAKAPACKER_NODE_MODULES_BIN_PATH"] || `yarn bin`.chomp

Since Yarn Berry does not return a valid path, the variable @node_modules_bin_path does not contain the node_modules/.bin path.

A potential workaround (while still allowing the yarn run webpack through the package.json) is currently to provide the node_modules/.bin path through the SHAKAPACKER_NODE_MODULES_BIN_PATH environment variable. Still, I would consider the current behaviour to be a defect of Shakapacker, potentially caused by the changed behaviour of yarn bin.

Setup environment:

Ruby version: 3.2.2
Rails version: 7.0.8 / 7.1.1
Shakapacker version: 7.1.0

So... we need a path to a single binary that we know will be there (webpack) and then we cut off the binary filename, yeah?

Seems like that would work for both Yarn Classic & Yarn Berry.

As for yarn bin webpack not working correctly with Yarn Berry, please open an issue at https://github.com/webpack/webpack

@MrSerth would you mind retrying using the soon-to-be-released experimental support for general js package managers? aka #349 - you should be able to enable it by ensuring SHAKAPACKER_USE_PACKAGE_JSON_GEM=true is set in your env (you can add this to bin/shakapacker and bin/shakapacker-dev-server).

I expect that to give you the most "native" results - it might still be possible to have an inf. loop but I would expect at that point it won't be something Shakapacker can/should be handling since the idea now is to defer to the package managers to handle invoking binaries (which also means what I think you're describing @Judahmeek would be a no-go)

Thanks for your responses!

I just went ahead and upgraded Shakapacker to the latest version available on the master branch (https://github.com/MrSerth/shakapacker-yarn-4/commit/c64bcc4ec23eee06653d5e854aac8796903d7164). Without setting the environment variable, the erroneous behaviour still exists:

shakapacker/lib/shakapacker/runner.rb

Lines 30 to 34 in 91f7fbe

    
           def fetch_node_modules_bin_path 
        
             return nil if Shakapacker::Utils::Misc.use_package_json_gem 
        
             ENV["SHAKAPACKER_NODE_MODULES_BIN_PATH"] || `yarn bin`.chomp 
        
           end

With Yarn Berry, the return value of the fetch_node_modules_bin_path is still no path.

Nevertheless, specifying the environment variable you suggested (https://github.com/MrSerth/shakapacker-yarn-4/commit/8f01580351e9e712b00d777c10f5723b102489c4) changes the behaviour, so that fetch_node_modules_bin_path always returns nil(as expected), avoiding the problematic call to yarn bin. As a result, my workflow is working again without any issues.

Do you have plans to default to using the package_json gem in the future (or otherwise tackle the changed output of yarn bin)?

Awesome! Yup the plan is hopefully that'll be the way shakapacker handles dealing with package managers - for now though it's feature flagged just so we can get some people testing before making it the default since that'd technically be a breaking change.

It would be great if you could start using that and let us know how it goes! (We should have a new release out in the next couple of days too so you don't need to depend on the git repo)

Alright, that's great to hear! I'll enable the flag on our repositories to try it further and will let you know in case I encounter any issues. Thanks again for your work 🙌

@MrSerth 7.2.0-rc.0 has just been released so you can switch from the pulling the git repo if you like :)

Awesome, thank you! I've upgrade to this pre-release in our projects using Shakapacker and successfully deployed the change to production.

	def fetch_node_modules_bin_path
	return nil if Shakapacker::Utils::Misc.use_package_json_gem

	ENV["SHAKAPACKER_NODE_MODULES_BIN_PATH"] \|\| `yarn bin`.chomp
	end