Yarn's Future - v2 and beyond

Question

Yarn's Future - v2 and beyond

arcanis opened this issue 6 years ago · 56 comments

Hi everyone,

When the Yarn project started back in 2016, our landscape was very different from what it is now. Package locking was far from being a first class citizen in the Javascript ecosystem, and the time needed to run an install was ... well, it was what it was. Yarn's release shook the status quo and started a movement that ended up being beneficial to everyone, other package managers included. Now is the time to assess where we are, reinforce our strengths and patch our weaknesses.

This thread aims to expose our roadmap for the next major Yarn release, and let you know about significant changes that we plan to make regarding Yarn's design. The codename for these changes is Berry - that's how we'll refer to it during the next few months.

Roadmap

This section lists some of the work we have in progress. It's far from being a comprehensive list, but it gives a good idea of the kind of changes you can expect going forward:

Major changes

The lockfile (and configuration) format will become a strict subset of YAML. In the process, the lockfile information will be slightly changed to account for some long-awaited changes (such as #5892).
We'll add support for plugins, which will be able to alter various things - from adding new commands to hooking into the resolution / fetching / linking steps to add support for new package sources or install targets.
Related to the plugin system, Yarn will become an API as much as a CLI. You can expect to be able to require it and start using its components in your script - no need to parse your package.json anymore, no need to run the resolution .. Yarn will abstract all those tedious tasks away.
Support for both Node 4 and Node 6 will be dropped. We don't expect Berry to have stable releases until after Node 6 gets EOL (expected April 2019), by which point it won't matter anymore.
The log system will be overhauled - one thing in particular we'll take from Typescript are diagnostic error codes. Each error, warning, and sometimes notice will be given a unique code that will be documented - with explanations to help you understand how to unblock yourself.
Some features currently in the core (such as autoclean) will be moved into contrib plugins. They'll still be supported, but might have a different release cycle than the standard bundle.
The codebase will be ported from Flow to TypeScript. To understand the rational please continue reading, but a quick summary is that we hope this will help our community ramp up on Yarn, and will help you build awesome new features on top of it.
The cache file format will switch from Tar to Zip, which offer better characteristics in terms of random access.

New features

Nested workspaces (when one of your workspaces itself defines additional workspaces) will be supported out of the box. This will open up new exciting usages, especially when used together with git submodules.
Running yarn run foo:bar (note the :) will run the foo:bar script regardless of which workspace implements it. This will make working with monorepos a lot easier, as you won't have to constantly cd between directories anymore.
Running yarn ./packages/my-package add foo will run yarn add foo into my-package. This should also contribute to make monorepos more pleasant, as you won't have to cd to add a new dependency somewhere.
A new resolution protocol, workspaces:, allows you to force the package manager to link one of your packages against a workspace. This makes it impossible to accidentally use a version of a package obtained from the remote registry.
A new command will appear, yarn constraints. This command will allow you to enforce constraints across workspaces - for example to require all your declared dependencies to use a unique version of a package.
A new type of dependencies will be introduced, portals. Portals are like link:, but follow the chain of dependencies (whereas link: dependencies never have any dependencies, but don't require a package.json either).
The yarn link command will now persist its changes into the package.json files (through link: dependencies), and the linked packages will now properly resolve peer dependencies (this feature is brought to you by Plug'n'Play 😉).
Writing posix command lines inside your scripts field will work regardless of the underlying operating system. This is because Berry will ship with a portable posix-like light shell that'll be used by default.
Scripts will be able to put their arguments anywhere in the command-line (and repeat them if needed) using $@. Similarly, scripts will have access to $1, $2, etc.
The PnP hook will be able to read files directly from the cache Zip archives - not unlike the phar archives in php.
Our cache will finally become fully atomic - multiple Yarn instances will be able to run concurrently on the same cache without risking corrupting the data.

And of course, a large amount of bugs are also expected to disappear as we modernize the codebase. A lot of those were caused by intricacies in the various components making up Yarn and, by cleanly separating them into standalone steps, we expect the overall logic to become much clearer and less error prone.

As you might guess, this isn't a comprehensive list - we plan Berry to be the big refactoring that Yarn had needed for a while now so many internal systems will be reworked. After that we expect following majors to become much smaller, as we'll have strong foundation for building new features and reworking the project one component at a time.

Yarn's philosophy

Now that we've discussed about most of the "What", let's discuss about the "Why". First, those are the values we always believed in and around which we continue to design most of our features.

Developing Javascript projects shouldn't leave the door open to surprises

This is our number one motto. We believe that Yarn should have a totally predictable and reproducible behavior. That your package manager has a responsibility to warn you when your project relies on unsafe patterns that could jeopardize your application at the worst possible time.

Developing Javascript projects should be easy

The complexity of the average Javascript project grown during the past years in order to accommodate with the ever increasing number of use cases people found. We believe that the package managers are in a unique position to help decrease some of the cognitive load and make it easy for their users to manage their projects and follow good practices.

Contributing to the Yarn ecosystem should be simple

Because a single project will never be able to satisfy all the needs from all developers counting on us, various projects spawned and grown, reinforcing Yarn rather than competing against it. Some examples are Create-react-app, Lerna, OpenCollective, or patch-package. We like that approach and want to give you the tools to do even more in a more efficient way. Not all features belong to the core, but being able to innovate and experiment is at the core of our philosophy, and we really mean it.

Goals for 2019

Now that our values are clear, what are the next steps we need to do in order to reach our goal? Note that some are long-term, some short-term, and that all are open to discussion (ideally in separate threads, otherwise it might get noisy 🤡).

Yarn will become a development-first tool

We've been saying it for some time now, but now we're ready to start going for it. Package managers are not tools you should ever run on your production servers. The more code run there, the higher are the probabilities that something will eventually go wrong and bring your production systems down. Yarn being development-first means that we'll make it possible for you to reach the state where you clone a repository, and voilà. This includes putting the emphasis on Plug'n'Play, which has been designed with this specific use case in mind.

Yarn will be rewritten in TypeScript

Yarn is currently fully covered by Flow types, and to be honest it works pretty well for us - the key part being “us”. Because we want to make it as easy as possible for third-party contributors to shim in and help us maintain this awesome tool, we'll switch the codebase to TypeScript. We hope this will help making the codebase feel more familiar than the projects your already contribute to.

Yarn will become an API, and its internal components will be split into modular entities

This one is huge. At the moment, when working with Yarn, your only option is the command line interface. We don't offer primitives that allow you to take advantage of the complex logic we've implemented - whether it's the resolvers, linkers, or even accessing the configuration. This will change, and Yarn will be redesign to become an API first and a CLI second.

Yarn will be designed to support different install targets than Node

Package management is a problem that keeps on being reinvented - even Yarn itself did it in the past. We believe the reason for this is that all package managers need to layout the installed packages in slightly different ways in order for them to be read by the host. This unfortunately is rarely possible, and it ends up being easier to rewrite the package manager and discard the already-existing user experience and features. Starting from Berry, we made it an explicit goal that each component of our pipeline can be switched to adapt to different install targets. In a way, Yarn will now be a package manager platform as much as a package manager. If you're interested into implementing PHP, Python, Ruby package installers without ever leaving Yarn, please open an issue and we'll help you get started!

Overall compatibility will be preserved when possible

Such semver-major changes are by design backward-incompatible, but we'll make sure to keep them to an acceptable level. In particular, the core command (yarn install, yarn add, yarn remove, yarn run) will keep the same behavior, and your previous yarn.lock will be silently migrated. One important caveat: our installs will now use Plug'n'Play by default. We've proven during the past few months that the approach was sound, worked to trim all potential implementation issues, and finally discussed with project maintainers to figure out whether there was anything we could do to help them make sure everything was ready. Now is the time to take the plunge.

FAQ

When will it be released?

We'll open a new repository early February with the v2, the time to flesh out some remaining interfaces. Starting from then, we'll make Berry accessible simply by running yarn policies set-version berry within your project.

What will happen to the v1?

As mentioned before, Berry won't have a stable release until this summer, and work will continue on the v1 until then. Even after Berry comes live, the v1 branch will still stay well maintained for the foreseeable future, perhaps distributed through a codename. This will ensure that your applications that work now continue to work in the future, until you feel ready to move to the v2.

How can i help?

The biggest way you could help us at the moment would be to enable PnP (yarn --pnp) and report any issue to the maintainers of the packages that don't work yet (ideally with a PR if you have the time). Most of the work they would need to do is to make sure they use the standard require.resolve api, or offer a way to configure a different resolution strategy. At the moment the biggest blocker we have is the tsc default CLI, which doesn't do either (thankfully ts-loader is just fine - we actually use it in order to compile our bundles!).

Answer 1 · 2019-01-24T15:12:35.000Z

[*]: Lack of tsc support also affects code completion in VS Code, as it uses the same code as tsc for looking them up.

It may appear to work out of the box sometimes but that's only for types published in DefinitelyTyped which are fetched in the background by the IDE.

This small bit of negativity aside, really looking forward to the future, in particular to better workspace workflows 🎉

And, who knows, tsc support for pnp 😇

Answer 2 · 2019-01-24T16:01:48.000Z

Love the idea of plugins. Use cases I have are filtering out dependencies that should not be installed (by looking at their package.json metadata or files list) and filtering out files that aren't needed and should not be extracted to disk.

Answer 3 · 2019-01-24T16:10:51.000Z

Writing posix command lines inside your scripts field will work regardless of the underlying operating system. This is because Berry will ship with a portable posix-like light shell that'll be used by default.

That sounds promising. Does that mean no more need for things like cross-env to be able to do MY_ENV=foo some-command?

Answer 4 · 2019-01-24T16:13:37.000Z

Question!

The lockfile (and configuration) format will become a strict subset of YAML.

Yarn will be designed to support different install targets than Node [...]
Starting from Berry, we made it an explicit goal that each component of our pipeline can be switched to adapt to different install targets. In a way, Yarn will now be a package manager platform as much as a package manager. If you're interested into implementing PHP, Python, Ruby package installers without ever leaving Yarn, please open an issue and we'll help you get started!

Does this also mean that the yarn.lock is going to be optional, and Yarn would be able to (with the help of plugins) read and write to (for example) package-lock.json and in the future composer.lock?

Or will Yarn still depend on the yarn.lock file to save the exact state of the Node / PHP dependency tree?

It would be awesome if Yarn can be used on projects built for NPM.

Answer 5 · 2019-01-24T16:24:05.000Z

Ah ah, good question - no (mostly), and yes (but not really) 🙂

The new architecture will see its components moved into various packages - for example plugin-http will only contain the code needed to resolve and fetch the packages when referenced through https?: URLs. Those plugins will all use the API interfaces exposed by the core package in order to work properly - and because (at least at the beginning) the core package will handle the generic resolution algorithm, it will be its task to load the lockfile.

The core package will not support anything else than yarn.lock. We have to put the line somewhere, and given that we don't want people to use Yarn and npm on the same projects (use any of them, but stay consistent) we won't make efforts to support this use case. Additionally, the npm lockfile has characteristics that simply don't make it suitable for us - and the other way around. We've actually discussed it before (#5654).

That being said, I guess you could reimplement the core and read from the package-lock.json file. It's just that it wouldn't map well with the design of the engine, and would (imo) be counter-productive. Again, use a single version of a single package manager in a single project - that's the only way you can ensure reproducible results and get rid of "work for me".

Answer 6 · 2019-01-24T16:27:08.000Z

The yarn link command will now persist its changes into the package.json files (through link: dependencies), and the linked packages will now properly resolve peer dependencies (this feature is brought to you by Plug'n'Play 😉).

So when a package will be linked into a project, even if the package finds it's dependency in it's surrounding before finding it in the project, it will use the peerDependendy provided by the project anyway? Ref: #6888

Does it means it will be required to use Yarn PnP to use this feature ?

Anyway, thanks for the share this list is awesome!

Answer 7 · 2019-01-24T16:29:36.000Z

So when a package will be linked into a project, even if the package finds it's dependency in it's surrounding before finding it in the project, it will use the peerDependendy provided by the project anyway?

Yep.

Does it means it will be required to use Yarn PnP to use this feature ?

Yes, this is only possible because PnP is aware of the full dependency tree (when it would otherwise be Node that would try to locate the link dependencies, based on the realpath).

Answer 8 · 2019-01-24T16:30:55.000Z

Again, use a single version of a single package manager in a single project - that's the only way you can ensure reproducible results and get rid of "work for me".

@arcanis Stepping away from the "using the same package-lock.json file" question; was it considered to use the same format as npm's package lock? Or a superset of it?

I really like yarn.lock becoming programmatically accessible as YAML, but using the same format would make it easier to read both using the same tooling.

Answer 9 · 2019-01-24T16:34:30.000Z

was it considered to use the same format as npm's package lock? Or a superset of it?

Yes, but as I mentioned the npm format has (what we believe are) design flaws that make it incompatible with our goals. We've already got this discussion in the linked thread 😉

Answer 10 · 2019-01-24T16:40:42.000Z

Can't wait!

Answer 11 · 2019-01-24T16:42:29.000Z

That being said, I guess you could reimplement the core and read from the package-lock.json file. It's just that it wouldn't map well with the design of the engine, and would (imo) be counter-productive. Again, use a single version of a single package manager in a single project - that's the only way you can ensure reproducible results and get rid of "work for me".

I understand completely that it can lead to unexpected results, but as long as its logic can be overwritten with plugins, it can be easier to migrate and convince others to use Yarn on already existing NPM and Composer projects. But of course, only with the help of plugins; this helps keep the core stable.

Answer 12 · 2019-01-24T16:48:57.000Z

Is there any way to add npm’s audit feature to yarn? IMO, that is a huge step toward better security, and I would love to have inside yarn.

Answer 13 · 2019-01-24T16:50:05.000Z

Yarn already implements audit since the 1.12 thanks to @rally25rs's efforts 🙂

Answer 14 · 2019-01-24T18:00:51.000Z

Yarn will be designed to support different install targets than Node

Hoping for solid interop with Rust crates!

Answer 15 · 2019-01-24T18:03:00.000Z

Yarn will be designed to support different install targets than Node

Really cheering for that one!

Answer 16 · 2019-01-24T18:31:17.000Z

Would love a future writeup on the Flow=>Typescript conversion from the perspective of Yarn - difficulties/ease, time involved, costs, benefits, etc.

Answer 17 · 2019-01-24T19:23:08.000Z

What will the installation story of yarn look like?

If there are different plugins and yarn targeting platforms outside of Node.js.

Answer 18 · 2019-01-24T23:17:32.000Z

totally interested to see where this goes in terms of deployments without running yarn install on production servers...thanks for setting this topic up @arcanis

Answer 19 · 2019-01-24T23:46:31.000Z

Yarn will be rewritten in TypeScript

🎉

Answer 20 · 2019-01-25T01:45:32.000Z

This is great.

Yarn will become an API, and its internal components will be split into modular entities

This is by far the best thing IMO especially combined with plugins.

Yarn workspaces transformed how I work with javascript. With an API and the ability to write plugins, I can imagine workspaces being made a lot smarter.

Here is a common problem I run into, and maybe there is a configuration based approach to fixing it but I've always relied on custom scripts

For example, a in workspaces project layout like

├── apps
│   ├── webapp-1/package.json
│   ├── webapp-2/package.json
├── components
│   ├── table/package.json
│   ├── form/package.json
│   ├── field/package.json
├── layouts
│   ├── collapsible-side-menu/package.json
│   ├── long-scrollable/package.json
├── themes
│   ├── light/package.json
│   ├── dark/package.json

If multiple teams or developers are working on this, there are some who are constantly touching layouts, and components, others who primarily work on themes, and others who build apps using these dependencies.

yarn workspace's behavior where it symlinks to the local project instead of the npm dependency is great, however it makes it so if I clone this repo from scratch and want to work on an app, I have to build all the themes, layouts, etc as well

i've worked around this with custom scripts, but they're hard to maintain, feel wrong, and involve a lot of shelling out to yarn to do work.

i think plugins + an api for yarn would make this so much cleaner.

Keep up the great work. This is one of the most exciting roadmaps i've seen in a while for a project.

Answer 21 · 2019-01-25T01:47:34.000Z

Why yaml for the lockfile? Presumably, the v1.x json lockfiles won't be compatible with the v2 yaml ones. Doesn't it make sense for it to be json for the native node support?

Maybe someone can explain the benefit, cause I'm not seeing it.

lockfile just jumped the 🦈

Answer 22 · 2019-01-25T02:09:49.000Z

i hope yarn will add yarn tasks , it can list all tasks in package.json scripts.

Answer 23 · 2019-01-25T02:38:03.000Z

Just a quick thought
echo Developing Javascript projects should be easy
Setting up test is a bit tedious regardless of it is jest or mocha...
Any ideas or thoughts incorporating test (should be the unit test) but allow flexibility also.

Answer 24 · 2019-01-25T07:23:03.000Z

@htoooth if I recall correctly, invoking yarn run without additional arguments serves this purpose

Answer 25 · 2019-01-25T07:42:43.000Z

Yarn will be designed to support different install targets than Node

Will yarn support deno soon?

Answer 26 · 2019-01-25T12:07:20.000Z

I hope yarn also support hjson or json5 as an alternative of YAML to maintain compatibility (look&feel) with package.json 😔

Answer 27 · 2019-01-25T12:33:34.000Z

To be crystal clear: the Yaml compatibility change only affects the lockfile (aka yarn.lock), not the package.json which will keep being the json it's always been. This change is just about adding a few missing characters that were making our lockfile format incompatible with Yaml unnecessarily 🙂

Presumably, the v1.x json lockfiles won't be compatible with the v2 yaml ones

The v2 will be able to read the v1 lockfiles.

What will the installation story of yarn look like?

The exact form is still to be defined - we'll likely end up with something similar to yarn plugin add ... that will download a plugin into a project-local folder, and Yarn will load them all from here.

Answer 28 · 2019-01-25T15:55:26.000Z

In a way, Yarn will now be a package manager platform as much as a package manager. If you're interested into implementing PHP, Python, Ruby package installers without ever leaving Yarn, please open an issue and we'll help you get started!

Am I reading this right? You want to replace composer? That's a very mature and feature rich tool right there.

Answer 29 · 2019-01-25T16:08:43.000Z

No, not really. We want Yarn's interface to be solid enough to allow for exploratorial work in this area, but we don't intend to spend much time on actually supporting those alternative install targets ourselves. Put differently, we simply want to make sure our codebase is clean and properly designed 🙂

Answer 30 · 2019-01-25T16:22:47.000Z

I think @3stacks asked a good question that I don't want lost in the noise: "Why yaml for the lockfile?"
Yaml is nice for human facing stuff, but it seems that the lockfile is primarily machine facing. That seems like a better candidate for JSON which has parsers in more languages than yaml. This seems relevant especially since you're thinking of opening the possibility of being a package manager for other languages.

Answer 31 · 2019-01-25T16:46:48.000Z

I don't want to generate too much noise, especially if this has already been discussed in the other thread, but I agree. Changing the format is a big deal and once it's decided, it's hard to go back on. If this is discussion better suited for a different ticket lmk.

The main advantage of JSON I see is that it's going to be readily available for parsing in any and all languages. Unlike JSON, Python for example doesn't have a yaml parser in the stdlib (and neither does JavaScript for that matter). Safe YAML parsers are also less common, and tend to have to be opted into. For what it's worth I'm pretty excited by the idea of a general use package manager and I'd be potentially interested in implementing pypi compatibility. I think safe YAML isn't a bad choice, but the more I think about it the more I agree that using JSON would be straight up better as lockfiles are not meant to be handwritten. Git diffs being slightly noisier are a very small price to pay for the benefits of being able to use simpler stdlib parsers, and not having to define the safe subset. I would also be curious to see if diff noise can be minimized somehow.

Answer 32 · 2019-01-25T17:01:46.000Z

Please open a new issue to discuss this particular topic, I would prefer not to focus the discussion from this thread on any specific point.

Answer 33 · 2019-01-26T09:57:28.000Z

Love the out of the box improvements for monorepos ♥️

Answer 34 · 2019-01-28T10:04:34.000Z

One of the biggest flaws with the way node_modules works is that modules like fs and various network native libraries (eg. https) are just open. Problem highlighted by drama in the community last year.

It would be nice if yarn could tackle this and have a "strict" mode that's very security conscious (eg. bubbles to the surface all modules that require disk, network and binaries for strict approval process). Understandably this can be tedious, but personally would find that acceptable.

I understand this potentially can be tackled with your plugin system, but I'm not a fan of something this sensitive (and with this many potential bells and whistles) be done under plugin limitations and as an after thought, rather then be a main line feature (just as a lockfile is).

Answer 35 · 2019-01-28T10:52:12.000Z

I think the ecosystem will eventually get there. However, to be truly effective, such a system would have to be implemented in Node itself imo - or at least some extra primitives would have to be exposed, such as an actual sandbox (vm isn't secure).

So not right now, but I know that there are efforts underway to make it happen, and from Yarn's perspective we would be open to integrate with them (for example by prompting for permissions at install time, etc).

Answer 36 · 2019-01-28T13:40:47.000Z

I know that there are efforts underway to make it happen

Oh wow, do you have links for this @arcanis? Sounds really cool.

Edit: Maybe some of the efforts mentioned are nodejs/node#24908 or nodejs/security-wg#327?

Answer 37 · 2019-01-28T14:42:10.000Z

@arcanis I'm also not sure if there is a process yet.

From @Trott nodejs/node#24908 (comment)

Node.js does surprisingly little with package.json. Most of the magic with package.json is in npm. There may be resistance to the idea that Node.js should parse and understand more package.json than it already does. A lot of feature requests come in the form of suggesting some package.json configuration and they are (as far as I've seen, anyway) always rejected.

I sincerely hope this doesn't end up in everyone waiting for everyone else.

I've managed to find issues and pull requests of nodejs devs trying out sandbox strategies but while sandboxing is tangentially related, it's functionally somewhat irrelevant (especially for things such as say network requests). Similarly, file system sandboxing may be considered safe with regard to nodejs runtime, but if we're thinking of it as "dependency rights" it doesn't matter if it technically it should-be-secure, from the perspective of an end user it's absolutely nonsensical to be there if it's unneeded. Not to mention that you may have cases where full system access is needed (I would imagine most of them).

To give an example, say there's a dependency that advertises itself as merging two specialized objects but requires file system access (sandboxed, for the sake of argument). Does file system sandboxing make a difference here? Because it doesn't to me, this requirement is too suspicious by itself.

I agree with you that nodejs features play a big part, but I think the registries and package managers are the ones who need to make the push forward and define at least some of what's required from a dependency tree stand point. Going by the comment by @Trott the nodejs team doesn't want to touch dependency tree problems.

Answer 38 · 2019-01-29T07:51:57.000Z

Still missing something like npm has audit. there is so many insecure modules around

Answer 39 · 2019-01-29T08:30:10.000Z

It would be nice if yarn could tackle this and have a "strict" mode that's very security conscious (eg. bubbles to the surface all modules that require disk, network and binaries for strict approval process). Understandably this can be tedious, but personally would find that acceptable.

@purpledrgn not sure if this is something that a package manager would tackle or closer to Node.js core. Keep in mind that this concern is not only for when you install a package but also for runtime. This topic has been raised a few times already in the issue queue and we have a PR opened for Node.js core to implement access control: nodejs/node#22112

Still missing something like npm has audit. there is so many insecure modules around

@vyteniskuciauskas-home24 checking your dependencies at install time is only a small part. There are more concerns around open source dependencies such as monitoring project deps after they've been deployed, and providing actionable remediation, etc. full disclosure, I work at snyk.io, so I can go on and on about all the benefits that you'd be missing out on :-)

Answer 40 · 2019-01-29T10:21:12.000Z

@lirantal

@purpledrgn not sure if this is something that a package manager would tackle or closer to Node.js core. Keep in mind that this concern is not only for when you install a package but also for runtime. This topic has been raised a few times already in the issue queue and we have a PR opened for Node.js core to implement access control: nodejs/node#22112

Can you elaborate? I've seen that before. Am I mistaken when I think it's simply a worker thread related feature?

I can see how this is similar to ACLs (which seems to be where it's clout is coming from) but I feel that's a very shaky road to go on: a lot of (potentially more verbose) complexity to get to the same result of telling packages what they can and can not do, with all the same problem areas (eg. a malicious express package would still have access to do malicious things). If the package managers forbid it via static analysis (ie. does it import 'fs' etc) or it's blocked at runtime, whats the difference aside from runtime variant being more costly?

For the idea in the issue to work (even assuming it can work with out separate threads) you would still have to have something akin to pseudo-runtimes for each dependency. And even then it still wouldn't solve any additional problems static analysis approach doesn't solve (assuming no flaws; which both impl can have)

The intent problem is only potentially solved via antivirus like pattern matching search; at least I don't see any other solution having come up in other areas with similar problems.

Personally I feel, given that a lot of packages people use actually do not need network or file access, a split between packages that require filesystem/network and everything else would solve a lot of issues. If you are a potential malware (due to your requirements) have your code signed-off by a trusted third party (or several). It would be interesting what the split is (among very popular packages) on packages that fall in either bucket. A quick peek at some of my own package.json files: database/cache drivers, 3rd party auth systems like auth0, http servers, network request libraries (eg. fetch implementations), some niche utilities related to ips, dev build deps that potentially have compiled or binary bits. Overall not a lot, and most from large groups, some backed or sponsored by large companies. At a glance I'd say no little to lose from a split and more rigorous "trust" process.

Answer 41 · 2019-01-29T11:17:41.000Z

Please open a separate thread for focused, more-than-one-answer-expected, discussions 🙏

Answer 42 · 2019-01-30T17:22:45.000Z

Question

Do you have any plans for yarn publish support for mono repos with workspaces?

Currently Lerna should be used to update all the packages versions and update dependencies. It would be great if yarn takes care of this too

Answer 43 · 2019-01-30T17:28:11.000Z

Yes and no. It's likely the publish workflow will be kept relatively similar to what currently exists now, but we'll be looking forward to see different implementations of it (through plugins). If an implementation becomes clearly superior, I could see us moving it into the core.

Answer 44 · 2019-02-06T17:53:25.000Z

Does this issue supersede #5773 (🦁 Yarn 2.0 Working Group ) - if so can we close it?

Answer 45 · 2019-02-12T15:02:00.000Z

Fantastic @arcanis, I tried to replace the https?: fetching to my own fetch-h2 (which handles HTTP/1(.1) and HTTP/2 transparently) to see if it'd be faster to use HTTP/2 (most public repositories handles it). This was surely painful, due to the implicit dependency of request, and its arguments being passed around the yarn codebase.

A plugin structure sounds great. I'd be happy to make a yarn-plugin-http2 (and if it proves to be fast, hand it over to you guys if you want it in the core).

Answer 46 · 2019-02-22T23:58:05.000Z

Update: I've published the trunk for the v2 on a separate repository (at least for now, in order to better triage issuers): https://github.com/yarnpkg/berry

It's already working mighty fine, but as you can guess we've still some way to go until we can call it a day. At the moment I'd say most of the remaining work are missing features (like outdated, upgrade-interactive, etc) and documentation, which are two things that anyone can help with! So if you want to have a strong positive impact on a project totalizing more than 34,000 stars on Github, look no further! 🙂

Answer 47 · 2019-02-24T19:24:22.000Z

What about using a more pnpm‑like system (i.e. preventing access to packages that aren’t declared in dependencies, devDependencies, peerDependencies or optionalDependencies).

Answer 48 · 2019-03-06T22:15:27.000Z

Do you have an anticipated release date yet for v2?

Answer 49 · 2019-03-06T22:50:03.000Z

When it's done! 😄

More seriously it's quite stable already - it can easily install itself, and I use it daily. There are still some important tasks that need to be done in order to start releasing it more widely, and any help we can get would be seriously impactful. I've listed various tasks in our issues and would be happy to give more details over there! Even documentation writers or early adopters would put us miles ahead 🙂

Overall, my target would be to have an official release somewhere this summer, but it might shift both ways depending on the progress and other external factors.

Answer 50 · 2019-03-23T09:00:17.000Z

Following thoughts are more of a feature request.

Right now, yarn is installed via package managers on a machine.

What about a portable yarn wrapper?

What's the problem I'm trying to solve?

hermecity of builds
reproducible builds

What's my current idea?

Check in an executabe file yarnw to version control repo.
Check in portable yarn/node version info in .yarnwrapperrc (filename tbd), example content could be yarn.version=1.15.2 \n node.version=10.15.3
Check out repo.
Run ./yarnw install, ./yarnw build (and bunch of other scripts from package.json)

Expected behaviour

$ ./yarnw install

Welcome to the portable yarn wrapper - best in town!

Checking ~/.yarn/wrapper/node_versions/10.15.3
Node 10.15.3 already installed. Ready to go!

Checking ~/.yarn/wrapper/yarn_versions/1.15.2
Downloading yarn 1.15.2 from https://yarnpkg.com/<version>.tar.gz
Downloaded.

Running yarn install (would execute ~/.yarn/wrapper/node_versions/10.15.3/node -e "require('~/.yarn/wrapper/yarn_versions/1.15.2/libexec/lib/cli.js').default()" ???  )

yarn install v1.15.2
<usual yarn experience>
<...>
Stars Done in took-some-time.

Creating yarn wrapper

$ yarn wrapper
<create .yarnwrapperrc file in cwd>

Where I am inspired from?

Gradle wrapper: https://docs.gradle.org/current/userguide/gradle_wrapper.html

Answer 51 · 2019-04-06T11:15:55.000Z

@dherges I'm not sure, but I think this discussion may be interesting for you.

TBH, I'm strongly against such things like gradlew or yarn policies set-version.

Answer 52 · 2019-04-11T16:05:38.000Z

@arcanis

To be crystal clear: the Yaml compatibility change only affects the lockfile (aka yarn.lock), not the package.json which will keep being the json it's always been. This change is just about adding a few missing characters that were making our lockfile format incompatible with Yaml unnecessarily

Any chance at getting a JSON5-compatible package.json?

Project configurations would be much easier to maintain with features like comments, trailing commas, etc. There has definitely been user interest for these features (see: yarn/issues/1751 & npm/issues/3336), but the npm team rejected this request due to the personal preferences of a few members (even though the user community clearly disagrees).

I'd argue that improving the readability & maintainability of package.[json|yaml|whatever] is directly in line with Yarn's phillosophy > Developing Javascript projects should be easy.

Answer 53 · 2019-04-11T16:44:20.000Z

Additionally, please consider adding an option for providing an explicit path to package.json & node_modules.

The common practice of development utilities dumping their configuration & generated files into the project root is messy & obtrusive. It's fine to have that sort of thing set up as a default, but users should also have the option to maintain their project structure as they see fit.

For example:

./config/

    /jest/
        config.js

    /yarn/
        /node_modules/
            ...
        package.json

    /typescript/
        _common.json
        development.json
        production.json

    /webpack/
        _common.ts
        development.ts
        production.ts

./dist/
    ...

./src/
    ...

is much easier to navigate & maintain than:

./dist/
    ...

./node_modules/
    ...

./src/
    ...

./jest.config.js
./package.json
./tsconfig._common.json
./tsconfig.development.json
./tsconfig.production.json
./webpack._common.ts
./webpack.development.ts
./webpack.production.ts

This would also eliminate the need for boilerplate .code-workspace files such as:

{
  "folders": [
    {
      "path": "C:/Path/To/Project",
      "name": "___  CONFIG  _________________",
    },
    {
      "path": "C:/Path/To/Project/src",
      "name": "___  SRC  ____________________",
    },
    {
      "path": "C:/Path/To/Project/dist",
      "name": "___  DIST  ___________________",
    },
  ],
  "settings": {
    "files.exclude": {
      "**/dist":         true,
      "**/src":          true,
      "**/node_modules": true,
    },
    "search.exclude": {
      "**/dist":         true,
      "**/src":          true,
      "**/node_modules": true,
    },
  },
}

Answer 54 · 2019-04-13T20:56:15.000Z

To quickly answer there are no plans at the moment to change the format of the package.json file. I believe it's current format is already too firmly rooted within the JS ecosystem for one project to be able to change it on their own - contrary to yarn.lock we aren't the only owner of this file. Yarn adding support for JSON5 would be more confusing than helpful if Node itself wasn't compatible with it, for example.

Answer 55 · 2019-04-13T20:57:28.000Z

I'm now going to close this thread as I believe I've answered most of the questions related to the plan outlined in the first post. Feature requests should be put into their own threads.

Note that the v2 implementation is going strong on the following repository: github.com/yarnpkg/berry, and we're always happy to receive feedback (and contributions!) from early adopters.

I hope to see you there! 😃

Answer 56 · 2020-01-24T17:58:56.000Z

Hi everyone! After a year of development we've just put Yarn 2 out of beta and into stable. For more information, please read the following post: https://dev.to/arcanis/introducing-yarn-2-4eh1

This is an exciting day - thanks to everyone who helped us on our journey 🌟🌟🌟