Lifecycle scripts for hosted git and local dependencies not run on install

Question

Lifecycle scripts for hosted git and local dependencies not run on install

rtsao opened this issue 7 years ago · 40 comments

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
When a hosted git dependency is installed, lifecycle scripts are not run (e.g. prepare)

If the current behavior is a bug, please provide the steps to reproduce.
Run yarn add user/repo or yarn add github:user/repo. No lifecycle scripts will be run.

What is the expected behavior?
prepare should be run on install (like with npm@5)

Please mention your node.js, yarn and operating system version.
Node.js 9.4.0
Yarn 1.3.2
MacOS 10.13

I've done some investigation as to why this broken. This is what I've found:

#3553 added the correct behavior, but only for git fetching, meaning the dependency was actually fetched via git.

Because Yarn recognizes certain git hosts, Yarn will install these dependencies via HTTP/tarball method instead of git (presumably for performance). So if this logic is triggered (which will happen with dependencies formatted like repo/user or github:repo/user, then the appropriate lifecycle scripts won't be run.

Related issues:

I'd be interested in making a PR to resolve this, but figured I'd make an issue first. I'm thinking that the logic for running lifecycle scripts in src/fetchers/git-fetcher.js should probably be moved into the more generic src/fetchers/base-fetcher, since per #3911 lifecycle scripts should be run for non-git dependencies as well.

Answer 1 · 2018-01-27T03:01:38.000Z

@rtsao I think I just ran into this issue, do you know if there's any workaround for it? Otherwise I was just going to vendor the code I wanted from a Github repo

Answer 2 · 2018-04-27T17:21:02.000Z

This bit me too. Here's my experience with this bug in case it helps anyone. My team used bower-away to migrate from bower to yarn, so all of our non-dev dependencies are under the @bower_components namespace. As we upgrade and add non-dev packages, we're trying to keep them all under @bower_components, for the sake of consistency.

These are the steps I took...

First try with the github repo and the namespace:

$ yarn add @bower_components/angular-ui-bootstrap@angular-ui/bootstrap#~2.5.6

$ ls node_modules/@bower_components/angular-ui-bootstrap/
CHANGELOG.md	    CONTRIBUTING.md  LICENSE	index.js       misc	     src
CODE_OF_CONDUCT.md  Gruntfile.js     README.md	karma.conf.js  package.json  template

The dist dir is missing due to this bug.

Now try with the github repo, but without the @bower_components namespace...

$ yarn add angular-ui/bootstrap#~2.5.6
$ ls node_modules/angular-ui-bootstrap/
CHANGELOG.md	    CONTRIBUTING.md  LICENSE	index.js       misc	     src
CODE_OF_CONDUCT.md  Gruntfile.js     README.md	karma.conf.js  package.json  template

The dist dir is still missing, again due to this bug.

Now try using package name, rather than github repo...

$ yarn add angular-ui-bootstrap#~2.5.6
$ ls node_modules/angular-ui-bootstrap/
CHANGELOG.md  LICENSE  README.md  dist	index.js  package.json	src  template

The dist dir is now there. However, I want it to be under @bower_components, since that's where all my other non-dev deps reside, so this doesn't work for me.

Try specifying package name instead of github repo, but using the namespace...

$ yarn add @bower_components/angular-ui-bootstrap@angular-ui-bootstrap#~2.5.6
error An unexpected error occurred: "http://npm-registry.my-company.com/@bower_components%2fangular-ui-bootstrap: Not found".

Error! Why doesn't this syntax work??? Is there no way to specify a package name along with a namespace??

I ended up working around this by installing using the package name without the @bower_components namespace, and then grabbing the resolved URL from yarn.lock, and running yarn add again using that:

$ yarn add @bower_components/angular-ui-bootstrap@http://npm-registry.my-company.com/angular-ui-bootstrap/-/angular-ui-bootstrap-2.5.6.tgz#23937322ec641a6fbee16498cc32452aa199e7c5

This works, but is obviously less than ideal, and I wasted over an hour on it.

Yarn maintainers, this bug has now been open for over 3 months. In addition, there are 7 related bugs, and 16 votes. Why has this not been addressed?

Answer 3 · 2018-04-27T17:41:34.000Z

Aliases are very experimental and we don't really recommend using them - even if they work, they break the expected semantics in various ways, especially with regard to peer dependencies. I would recommend you to find a way to use the packages from npm without using the @bower_components namespace.

Yarn maintainers, this bug has now been open for over 3 months. In addition, there are 7 related bugs, and 16 votes. Why has this not been addressed?

Mostly because I missed this issue, sorry @rtsao, you did a good investigation! Moving this logic to the base fetcher could work, yes. It would have to be called only in some cases though, since we don't want it to happen for regular tarballs (which are expected to already have been packed).

Answer 4 · 2018-06-19T16:19:16.000Z

Is this ever going to be addressed? I switch to Yarn because npm was falling behind on crucial features, but now it looks like the pendulum is swinging back again. This issue has been known in one ticket or another for probably 16 months now. Since at least version 0.23.x.

Answer 5 · 2018-06-19T21:18:57.000Z

Yarn is a community project. If a feature is so problematic, we invite you to contribute to fix it. Right now nobody stepped in yet, and it isn't something we are facing ourselves, so it's lower priority for us than other fixes and features. Still, if you open a PR, we'll gladly review it 🙂

Answer 6 · 2018-06-20T00:00:08.000Z

Totally understand if that's the holdup and don't get me wrong, I'm very thankful for the hard work! I would be happy to contribute when I find the time... Always the excuse I know.... My question was meant to be a, "Why isn't this a higher priority?" and not a "Why isn't this fixed dammit!?"

Here's why I think this is a big issue:

Github has a billion forks people definitely want to use as they would use a normal package. I avoid forking things quite a lot because npm and yarn both didn't use to manage this. It's actually what got me to try Yarn in the first place. Though it didn't have it either, I stayed for those sweet, sweet yarn.lock files.

I'm not sure if npm has addressed this either, I'm just surprised it's not a higher priority. Not being able to readily use both Github and local forks as expected massively increases iteration time and I feel like it must be a single line somewhere.

Can you possibly point me towards the part(s) of the codebase that manage invoking post-install scripts?

Answer 7 · 2018-06-20T11:23:10.000Z

Lifecycle scripts are executed here, which is called through here and here.

Why isn't this a higher priority?

Because no maintainer seem to have been suffering from this issue - it's not to say that the problem doesn't exist, but we all tend to work first on what affects us 🙂 We do try to fix problems other people report when we get the chance, but time is scarce and some help is always appreciated.

Answer 8 · 2018-06-20T15:18:26.000Z

Thanks very much @arcanis! I'll poke around in those files when I find some downtime and see if I can find a fix.

Answer 9 · 2019-01-09T21:13:40.000Z

Is there a recommended workaround for this? I'm using URLs like git+https://github.com/user/project.git in my package.json and hitting the same problem where prepare is not executed.

Answer 10 · 2019-01-09T21:53:37.000Z

My workaround is back to npm. I got too many private repos using prepare script, so I have to tolerant the slowness of npm, which is actually not too bad these days.

Answer 11 · 2019-01-10T08:55:19.000Z

I just use a postinstall in the main package.json, that cds in any git submodule I might have, and runs yarn in there too. Silly, but at least it works, until yarn actually works as expected :D

Answer 12 · 2019-01-22T20:09:02.000Z

I just hit a variant of this issue and it's a real pain.

I've got a library package that includes sources and a non-trivial build that I want to execute when the package is installed as a dependency. The runtime functionality supported by the package is not available to the dependent package until this build process completes.

I do not want the client (i.e. derived package that declares dependency on my lib package) to have any knowledge or direct coupling to the details of how the lib package is installed.

I believed that declaring a postinstall rule in my lib package was all that would be need to automatically execute my lib package build when it's installed.

But, unfortunately this lifecycle hook doesn't seem to ever get called iff the dependent package declares the dependency using one of the git syntax variants.

So this means that yarn install doesn't act consistently; the results are radically different based on how the package is installed.

Answer 13 · 2019-03-06T13:17:57.000Z

There's been a PR (#6131) to fix this open since August 2018. Is anything going to happen?

Answer 14 · 2019-03-17T19:52:42.000Z

I'm also having this issue, it would be great to see resolved. Has anyone found a workaround to force Yarn to use git+ssh dependencies as non hosted?

Answer 15 · 2019-04-05T17:50:43.000Z

We're also having this issue and I wanted to add flavor/motivation as to why: code review.

Assume we have two repositories: package-foo and app-bar.

app-bar is an application we're working on that depends on a custom package we wrote package-foo but needs new functionality implemented in package-foo in order to implement a new feature in app-bar. We want to be able to submit to PRs at the same time (one for each repo) for our internal code review process.

The issue here is if a developer submits a PR for app-bar that relies on unpublished modifications to package-foo (because those changes themselves are undergoing code review in their repository) there's no way to indicate in app-bar/package.json that the changes rely on new functionality implemented in an unpublished commit of package-foo w/o using git URIs and this bug means this process doesn't work if package-foo is written in typescript or has any build process defined before it's require-able at runtime.

Our workaround in the meantime is that we don't submit a PR for app-bar until package-foo's PR is merged and published. Then we submit a PR with the actual npm version of the package referenced in app-bar. Which might be the "more correct" way to do this since we'd have to change them to version numbers before merging anyway.

Answer 16 · 2019-04-11T10:08:16.000Z

Hi, this issue is wired but this is what worked in my case:

I have a GitHub repo, with this "package.json script" which runs babel (for some "jsx" transforms):
"scripts": { "prepare": "babel src --out-dir ." }

And I reference as a dependency in my project like this:
"@namespace/repo-name": "github:namespace/repo-name"

Now I can run yarn and it usually runs the "prepare" script correctly, but sometimes remove the yarn.lock file is necessary.

Answer 17 · 2019-04-16T05:18:08.000Z

The approach from @zicrox seems working. But if I add my build directory to .gitignore, yarn will still skip the prepare script (I have no clue how these two are connected). Hopefully someone would confirm this.

Answer 18 · 2019-05-07T10:06:49.000Z

@qinsoon I managed to get mine working by adding my dist folder to the include in package.json:

Slice of package.json:

  "main": "dist/index.js",
  "files": [
    "dist/"
  ],
  "scripts": {
    "clean": "rm -rf ./dist",
    "build": "tsc",

slice of .gitignore:

**/node_modules/
yarn-error.log
dist

Hopefully that's helpful...

Answer 19 · 2019-06-18T11:13:26.000Z

I managed to get mine working by adding my dist folder to the include in package.json

This right here. Thanks @toshi38

Answer 20 · 2019-09-21T11:56:52.000Z

Same issue for local dependency like file:/Users/user/something

Answer 21 · 2019-10-15T03:28:41.000Z

Another one here who was confused on why I wasn't seeing any build files on the installed package.

Answer 22 · 2020-01-06T16:26:13.000Z

In my case the prepare script was executed but the build artefacts were deleted by Yarn because they matched the .gitignore. I solved this by adding an empty .npmignore, which Yarn picks over the .gitignore.

Answer 23 · 2020-01-14T21:51:16.000Z

@njlr bless you for i was already starting to get a real headache before seeing your comment.

Answer 24 · 2020-05-12T19:17:02.000Z

I came to this issue with a typescript package, I made a minimum working example here: https://github.com/richard-ejem/tspackage

See my SO question https://stackoverflow.com/q/61754026/1660584 (and my own answer to it) - it is weird that yarn keeps index.js in the dist folder, therefore the prepare script is not skipped, just its results are removed, except of index.js for unknown reason.

It also shows a difference in npm and yarn behavior, npm does not remove files created by prepare script at all.

Answer 25 · 2020-08-28T15:54:36.000Z

I've added the empty .npmignore file but prepare output is still not present so I think that the prepare script is not being run... yarn version 1.12.3. npm install <my-git-repo> works fine.

Answer 26 · 2020-08-28T18:53:13.000Z

I've added the empty .npmignore file but prepare output is still not present so I think that the prepare script is not being run... yarn version 1.12.3. npm install <my-git-repo> works fine.

That's pretty old version, have you tried current?

Answer 27 · 2020-08-28T20:32:36.000Z

I've just upgraded to v1.22.5 and still no success.

Answer 28 · 2020-08-31T14:56:38.000Z

Also present in v2

Answer 29 · 2020-08-31T15:04:49.000Z

@KishanBagaria If you use the prepack script as documented, it'll work just fine in Yarn 2.

Answer 30 · 2020-09-01T14:49:46.000Z

@arcanis I'm not going to pick a yarn specific solution; doesn't feel very futureproof, a solution that's compatible with other package managers is preferable. A perfect example for this would be if I'm running a build and launch of an up within a slim docker container running node; I'm not gonna bother having yarn as a dependency for that.

However, thanks for your assistance regardless!

Answer 31 · 2020-09-01T14:51:57.000Z

I don't know what you mean, prepack has been a standard script for ten years. It's nowhere Yarn specific.

Answer 32 · 2020-09-01T14:57:42.000Z

Ah, my bad!

Answer 33 · 2021-01-06T00:41:58.000Z

There has been an open PR for this for 2 years! How is this still a problem? I've been using and loving yarn for years, but this one bit me and cost ALOT of time

Answer 34 · 2021-01-06T00:51:13.000Z

This issue is fixed in Yarn 2+, and the fix will not be backported in 1.x. Please check the Migration guide and continue from there.

Answer 35 · 2021-01-06T20:35:57.000Z

This issue is fixed in Yarn 2+, and the fix will not be backported in 1.x. Please check the Migration guide and continue from there.

Yep, and Yarn 2.0 is why I stopped using Yarn!

Answer 36 · 2021-09-28T00:26:46.000Z

@arcanis I don't see prepare mentioned anywhere at https://yarnpkg.com/advanced/lifecycle-scripts. Is it supposed to be mentioned there?

Answer 37 · 2021-10-09T03:05:04.000Z

@arcanis I don't see prepare mentioned anywhere at https://yarnpkg.com/advanced/lifecycle-scripts. Is it supposed to be mentioned there?

According to the Yarn Rulebook Yarn 2 uses prepack instead.

Answer 38 · 2021-10-09T04:18:44.000Z

@arcanis I don't see prepare mentioned anywhere at https://yarnpkg.com/advanced/lifecycle-scripts. Is it supposed to be mentioned there?

According to the Yarn Rulebook Yarn 2 uses prepack instead.

Prepack is not the same. They’re run at different times.

Prepack is run before packing. Not after installing unlike prepare.

Answer 39 · 2021-10-09T04:45:51.000Z

@arcanis I don't see prepare mentioned anywhere at https://yarnpkg.com/advanced/lifecycle-scripts. Is it supposed to be mentioned there?

According to the Yarn Rulebook Yarn 2 uses prepack instead.

Prepack is not the same. They’re run at different times.

Prepack is run before packing. Not after installing unlike prepare.

Yes, in npm prepack and prepare are run at different times.

But in Yarn 2 prepare is deprecated and never run (according to the Yarn Rulebook). Instead, it is prepack that is run both before packing and after installing (but only for Git dependencies, since Yarn first fetches the repository and builds it using yarn pack, according to the Lifecycle Scripts).

Answer 40 · 2022-01-14T20:46:19.000Z

Any workarounds for v1 users?