Doesn't work with dependency of dependency
Worthaboutapig opened this issue · 53 comments
Supposing I have a main project A which has a dependency on Project-B, which in turn has a dependency on Project-C. Both Project-B and Project-C are managed as yalc dependencies.
npm install throws an error such as:
Could not install from ".yalc/project-b/.yalc/project-c" as it does not contain a package.json file.
When installing the reference to project-c, the project-b .yalc folder is the same as usual file:.yalc/project-c- however, this is then attempting to reference the dependent project within the project copied to the .yalc folder, however, it should be referencing the yalc folder from the root of the .yalc in project-b instead.
Basically, if you have a project in your .yalc folder that references another project in the .yalc repository, then it will incorrectly assume the referenced project is at the same level, when it should be looking to the root yalc folder in the parent project (i.e. project-a/.yalc)
Could not install from ".yalc/project-b/.yalc/project-c" as it does not contain a package.json file.
Does .yalc/project-b/.yalc/project-c actually contains package.json? If no, I don't get why?
Hi, did you close this as I haven't replied? Sorry, I stopped using yalc as it didn't cover this case and I required it.
I could try and setup a test repo if I get the time, but basic file:// links are working ok for now. IIRC, then, no, there's no .yalc at all in the folder for .yalc/project-b. Should there be? Are all the projects copied recursively for every project that depends on yalc? So, if you have a lot of yalc dependencies to manage, there's be dozens of copies of the same projects in different subfolders of different projects in the .yalc folders? I presumed that there'd only be one of each, at the root, that the other projects will refer to.
.yalc at all in the folder for .yalc/project-b. Should there be?
Yes if you want yalced package project-c in project-b to be available when you install project-b as dependency of project-a then project-b/.yalc/package-c folder should be definitely included in project-b published code, even it is published and added via Yalc repo.
Yalc is actually very simple, it is not designed to handle some complex nested interdependencies of packages.
Ok, maybe yalc doesn't do what I'd expect. I'll try to find the time to create a demo repo and you can let me know whether I'm expecting the wrong thing.
So, if you have a lot of yalc dependencies to manage, there's be dozens of copies of the same projects in different subfolders of different projects in the .yalc folders? I presumed that there'd only be one of each, at the root, that the other projects will refer to.
You can just add all needed packages as yalc dependency for the root package. If you don't publish code in project's .yalc folder you should not include it in dependencies of package.json. So you manage all those dependencies explicitly. Yalc is actually designed to simplify development not complicate it. =)
This didn't work well for me. It means I have two versions of the package installed and deduping not working nicely with the TS packages I'm trying to fix for some reason.
@elie222 can you elaborate on you issue? you have problems with inner .yalc packages of the package published with yalc?
There is a bug currently in npm-packlist module that may not include package.json file of the inner folder in published content. Actually I've fixed it but not published yet.
Could not install from ".yalc/project-b/.yalc/project-c" as it does not contain a package.json file
This should not be the case now.
Well I tired using .yalc in a package that imports from another package.
But had problems with that, the same as others.
Well actually I don't really get what issues really are. If it is about absence of package.json in nested packages (folders), this should not be the case now. yalc is really very simple and transparent, there is no magic inside which could provide unexpected side effects.
@elie222 it depens what is meant by nested packages.
Say A package that you are going to publish/use with yal and B is package that you add to to A using yalc
So you will have
/ A (root of A package)
.yalc/
B/..
package.json <- this will probably contains in `dependencies`: `"B": "file:.yalc/B"`
So then you do yalc publish in A, it will publish A package which will contain .yalc/B folder and thus nested B package will be used.
Not sure what else is meant by "nested" packages.
I'm running into the same issue.
I have a project A depending on a project B which depends on a project C.
I ran:
cd ...C
yalc publish
cd ...B
yalc add C
yalc publish
cd ...A
yalc add B
yarn install --force
error Package "C" refers to a non-existing file '"...A/.yalc/B/.yalc/C"'.
I'm using the latest yalc version 1.0.0-pre.27.
I do not see any nested .yalc directory inside ...A/.yalc/B.
I did this with a pretty simple project setup, but let me know if you are unable to repro for some reason.
@whitecolor can we re-open this please? Looks like ppl are still having problems. See comment from @yang for reproducible steps. Thanks!
This is due to the relative path to the .yalc directory.
When yarn or npm try to install in A, it finds B's package.json in .yalc/B fine. But then B's package.json contains a dependency of file:.yalc/C. So it looks in .yalc/B/.yalc/C and finds no package.json.
If you manually change package.json in B after using yalc add C and change the entry to "C": "file:/full/path/to/B/.yalc/C", then when you run yalc publish in B and then yalc add B in A, things work as expected.
yarn/npm looks in .yalc/B/package.json, then finds /full/path/to/B/.yalc/C/package.json and can resolve everything OK.
I just tried:
cd ...C
yalc publish
cd ...B
yalc add C
yalc publish
cd ...A
yalc add B
yarn
I'm not sure how you're getting those results @whitecolor. I'm on OSX using v1.0.0-pre.31 and when I perform the same steps there is no nest_a/.yalc/nest_b/.yalc directory - my results match yangs results. Copying it over manually is manageable as a work around but I'd definitely like to know more about how that's working for you several other folks in this thread have been unable to get those results.
I believe I have the same problem with v1.0.0-pre32 (and previous versions).
in my case, I have two projects that the same dependency, A and B both require C, but also B requires A, so:
A has dependencies: C
B has dependencies: A, C
I am working on project C, but I have to test it using Project B. So...
From project C: build and yalc push
From project A: yalc add C, then build and yalc push
From project B: yalc add A C
and finally, in project B: yarn to install dependencies results in:
error Package "" refers to a non-existing file '"/Users/levi/Projects/B/.yalc/A/.yalc/C"'.
My only workaround is to manually update project B's package.json to use an absolute path to C within project B, e.g., "C": "file:/Users/levi/Projects/B/.yalc/C", and then run yalc push again from project C.
I will also chime in that I am experiencing this issue. It's an issue with the relative paths in the package.json. If yalc would use absolute paths with its file:/ adding, everything would work perfectly.
I solved my issue with this using resolutions.
"resolutions": {
"@ckeditor/ckeditor5-upload": "file:.yalc/@ckeditor/ckeditor5-upload"
}same problem here
I'd like to add that I recently encountered this issue as well.
Here's how I have things setup:
Project P
has packages [A, B, C]
A depends on [B, C]
B depends on C
C has no dependencies
Order of operations:
yalc publish Ccd ..Byalc add Cyalc publish Bcd ..Ayalc add C && yalc add Byarn=>error Package "C" refers to a non-existing file '"drive:\\A\\.yalc\\B\\.yalc\\C"'.
I was able to fix this by adding:
"resolutions: {
"C": "file:.yalc/C"
}
any updates on this issue?
any updates on this issue?
No, as the problem is not clear to me.
After trying this and other solutions for local modules I settled with Lerna, works with either NPM or Yarn
I settled with Lerna
Yalc can hardly be considered as an alternative to Lerna or alike tooling. It's just for quick grabbing a local (under development) package content and injecting it into your project(s). It tends to deal with simple and independent packages. When you have yalc dependencies inside the package you want to inject probably there may be some issues, but still, it should be all transparent and resolvable.
Problems with loading subdependencies can be hard to reproduce but as far as I remember one way to make it happen is to have a subdependency loaded in both root and another local dependency simultaneously
i.e. / - project folder
/package.json
-dependencies
local_dep1: "file: ./local_dep1'"
local_dep2: "file: ./local_dep2"
local_dep3: "file: ./local_dep3"
/local_dep1
/package.json
-dependencies
local_dep2: "file: ../local_dep2"
/local_dep2
/local_dep3
/package.json
-dependencies
local_dep2: "file: ../local_dep2"
Problems with loading subdependencies can be hard to reproduce
I wonder how yalc should solve those "problems"?
@wclr I believe I found one cause to this issue and am willing to create a PR if you're open to it.
one cause is if package.json -> files does not contain '.yalc' then nested dependencies breaks. This is due to the published repo not containing the '.yalc' directory
Here's a very simple reproduction
git clone git@github.com:olsonpm/yalc-nested-issue.git
cd yalc-nested-issue
./repro.sh
...
Error: Cannot find module 'b'
Require stack:
- /path/to/yalc-nested-issue/root/.yalc/a/index.jswhere the expected output is module a: a
For what it's worth, this is the main reason I created my "wrapper around yalc" that I named zalc: https://github.com/Venryx/zalc
Basically, it monkey-patches yalc so that when you run zalc publish/push, it adds the files in the being-published module's .yalc folder as part of its yalc-published contents (after adding a !.yalc/**/* line to the .yalcignore file); this allows a yalc-pushed module to contain a submodule's contents (in its .yalc folder), without those submodules becoming part of the npm-published contents as well (since that could cause bloat/redundancy/submodule-staleness).
While zalc has been working fine for me, it's not been tested enough to be "production ready" or anything; I mention it to spur ideas, and give an example of how yalc can be modified to include local copies of "dependency subtrees" rather than just a single dependency.
I can give more detailed instructions on how to set zalc up for this purpose is needed; zalc's entire source-code is only 41 lines long atm though, so that may be unnecessary: https://github.com/Venryx/zalc/blob/master/Source/Bin.ts
@olsonpm I checked the repro.
first, you need to include .yalc folder when publishing package a, this is done for example by adding .yalc to files in package.json
second, but this is actually the way npm works, don't know why it doesn't install file:.yalc/b that is in a's dependencies, if you use yarn or pnpm it will work out.
The strange thing though run npm twice (change npm i && npm i in repro.sh script) and it worked out for me -). So, seems something with npm (maybe buggy?) way of installing deps here.
why would users be expected to add '.yalc' to package.json -> files ? shouldn't yalc be responsible for that given it's required for nested dependencies to work ? and also you wouldn't want to publish yalc files to npm, it is a local dev tool.
I'll look into the npm i && npm i workaround you mentioned, that seems odd
why would users be expected to add '.yalc' to package.json -> files shouldn't yalc be responsible for that given it's required for nested dependencies to work ?
No, yalc totally mimics npm pack/publish behavior here, it is by design. If you wan't to publish such package to the npm you probably would want .yalc folder to be included there too.
Yalc works simple here, I don't see a reason to introduce some ad-hoc logic to handle nested yalc dependencies, this will only complicate things.
I don't see a reason to introduce some ad-hoc logic to handle nested yalc dependencies
the reason would be to support the use-case that a lot of people seem to have. It's understandable you don't want to support that use-case, but then I think a lot of us on this thread would prefer a fork which is totally fine. I enjoy working with this lib
the reason would be to support the use-case that a lot of people seem to have.
So what do you lack for the use-case? Is there a problem with including .yalc folder explicitly?
the problem is that yalc should handle that itself, it should not require the user to add '.yalc' to the list of npm published files. Users should not be publishing files to npm which are specifically meant for local development
A "compromise" would be to at least allow users to tell yalc they want to have the .yalc folder included in their yalc-publishes (without adding it to their npm publishes); my zalc wrapper does this by parsing !... entries in the .yalcignore file, and "adding back" those files to the yalc-publish. (so if you add !.yalc/**/* to .yalcignore, the .yalc folder will get reincluded in the yalc-publishes, by the zalc wrapper-code)
Currently, there is no way to tell yalc to respect the above entries, because of the way it parses the .yalcignore file. (it uses npm-packlist to get the initial entries, which does not respect the !... entries above -- and the subsequent parsing of the .yalcignore file is set up to only filter from that initial list so cannot "add back" the .yalc folder:
Line 155 in 12e3ef3
Users should not be publishing files to npm which are specifically meant for local development
I'm afraid this is not true, the fact that one uses yalc dependency in the package doesn't mean that it is strictly "for local development", one may want to publish it to the remote registry as well including .yalc content and deps. And we should try to keep consistency in terms of included content here.
the problem is that yalc should handle that itself, it should not require the user to add '.yalc' to the list of npm published files.
So I explained why it is not a valid point. Anything else preventing from adding .yalc explicitly to package content? (esp. if you are not going to publish to the remote registry)
Currently, there is no way to tell yalc to respect the above entries, because of the way it parses the .yalcignore file.
This too would make local publishing less consistent with remote (remote content should be >= local to prevent potential lack of files).
This too would make local publishing less consistent with remote (remote content should be >= local to prevent potential lack of files).
Yes, processing !xxx (ie. yalc-specific file-inclusion) entries in .yalcignore would make local publishing less consistent with remote, but only in the case where the developer specifically told yalc to do so. How is this a problem?
Usage of those entries would be:
- Opt-in (ie. the only people adding those
!xxxentries would be people who wanted yalc-specific file-inclusions) - Written in an already-existing yalc-specific
.yalcignorefile (so devs should already know that entries in this file will not affect the contents of "official npm publishes" to the global repository) - Make use of an already-existing
!xxxnotation that has been used in similar files (eg..gitignoreand.npmignore) - Enable a use-case for yalc that a fair number of users seek. (including a locally-modified dependency that itself includes a locally-modified subdependency is very useful, and something I do all the time; and from this thread [and others for npm/yarn], there seem to be quite a few that have need for this [or similar] functionality)
(If you're worried people will "accidentally" re-include files for local publishes only, when they actually were supposed to add those entries to the package.json files field [thus making it present in both local and remote publishes], then I think you are underestimating the cognitive ability of developers to know the difference. 😄)
I just currently fail to see why the existing behavior is not sufficient to cover the use case to make proposed changes (either auto include .yalc or do it via .yalcignore) valid/needed.
With the current behavior, there are two options for what to do with the .yalc folder for publishes:
- Never publish it -- neither for local yalc publishes, nor remote npm publishes.
- Always publish it -- for both local yalc publishes, and remote npm publishes. (well, or exclude it from yalc publishes only, but this doesn't help at all for my use-case)
What I and some others want (at least as an option):
3) Publish the .yalc folder, but only for local yalc publishes -- do not include it when publishing publicly using npm publish.
Why would we want option 3 rather than the existing two options?
Well, here are my reasons that option 1 and 2 are insufficient:
- Option 1 is insufficient, because it would be a pain to have to manually yalc-include both my "dependency B", as well as all the subdependencies of mine underneath it that I want to use the locally-modified versions of. I have a lot of custom npm packages (>30), and right now I "include" a large number of them in one go, by having them all as subdependencies under a "wrapper" package. This "wrapper" package is helpful because it "aligns" the versions of all my subdependencies together to a set that I know work well together -- both of my own packages, as well as packages multiple of them depend on (semver helps, but is not 100% infallible, as devs don't always follow it -- and even when they really try to, mistakes are pretty common, especially for Typescript typings and such). For this to work though (in a way that allows rapid local development), there needs to be a way for my yalc-included "wrapper" package to contain locally-modified versions of its subdependencies (ie. I want the wrapper package to be able to retain control over the versioning of the subdependencies, so all projects using the wrapper package use the same set, while using yalc at both dependency and subdependency levels).
- Option 2 is undesirable, because it would bloat and non-standardize the "public releases" of my "wrapper package", by duplicating the subdependency resolution/inclusion process that is provided by npm/yarn. For public releases (ie. once I'm done with my set of rapid local changes), npm/yarn's regular release and dependency-resolution process works fine -- and is preferred, since it's more common/standard than yalc. In other words, I want the "yalc" process to be something that only I as the developer of the wrapper package (and its subdependencies) uses as a shortcut for faster local development, but I do not want that local shortcut tool to impact the way that end-users consume my packages once I'm ready to release the official public versions.
I'm not proposing to include not-needed content when publishing to remote. If you want to publish without .yalc, do you still have yalc deps in package.json?
If you want to publish without .yalc, do you still have yalc deps in package.json?
No; what I do is I have package.json list the last "npm-published version" for its subdependencies, and then I use this:
"workspaces": [
".yalc/*"
],
What this accomplishes: When yalc is used locally, Yarn notices the folders in .yalc and uses that rather than retrieving the npm-published contents. This lets me make local modifications to my subdependencies (which is managed by yalc), without me having to break/"lock to using yalc" the package.json file that is pushed to npm. (when my package is installed by other devs not using yalc, the .yalc folder contains no modules, so no overriding from the npm-published versions of them occurs)
i suppose we'll see how many people that use this lib feel comfortable publishing content intended for local development out to the public registry. i think the yalc files have no place there and i feel others with this use case will feel similarly. i will use my fork as a result
No; what I do is I have package.json
If you package.json doesn't have any references to packages located in file:.yalc, then I don't get why you would want .yalc folder included in the package content at all?
i suppose we'll see how many people that use this lib feel comfortable publishing content intended for local development out to the public registry.
That depends on your perspective for what it is intended. What your fork is going to do, include .yalc while (local) publishing by default?
If you package.json doesn't have any references to packages located in file:.yalc, then I don't get why you would want .yalc folder included in the package content at all?
It doesn't have references hard-coded into the package.json (ie. in the dependencies section), but the .yalc folder contents are still used as the source of the corresponding node_modules contents, because of this part:
"workspaces": [
".yalc/*"
],
It auto-links the modules in the .yalc folder into node_modules. (it's a cool little "repurposing" of Yarn's workspaces feature)
So it achieves "user-specific yalc-inclusions". (ie. including the modules that the user did yalc add for [for example, the ones I add locally as the developer], without having to modify the package.json file in a way that makes npm/yarn resolution/including break)
it's a cool little "repurposing" of Yarn's workspaces feature
I use yarn/pnpm workspaces with yalced package the same way in my monorepo projects. Are you publishing (locally using yalc) this package that has "workspaces" inside of it? I don't get why, and how it is supposed to be used?
Are you publishing (locally using yalc) this package that has "workspaces" inside of it?
Yes.
I don't get why
Because that's the smoothest way I know to have the packages in .yalc symlinked into node_modules without having to "overwrite" the regular version numbers for those dependencies in the package.json file. (the other ways are more disruptive to people trying to use the same module as a "regular npm package")
and how it is supposed to be used?
It is used to have Yarn symlink to a locally-modified version of the subdependencies (the copy in the shallow dependency's .yalc folder), rather than requiring pushing an npm update for every little change I make, when developing. (and doing so as a bundle, rather than having to manually yalc-add every subdependency in every root project)
I'm not sure what part of the explanation is missing? (If you mean, how do I tell yarn to do that symlinking -- it does it automatically. Perhaps it's not supposed to? But anyway, it does, because I'm currently using it this way in multiple projects.)
Because that's the smoothest way I know to have the packages in .yalc symlinked into node_modules
I'm not asking about this, this is how yarn workspaces work inside a project.
What I'm asking:
In your project A you have .yalc dir, and package.json with workspaces settings (which point to package from .yaclc), and then you publish this A with yalc then to add into another project B?
In your project A you have .yalc dir, and package.json with workspaces settings (which point to package from .yaclc), and then you publish this A with yalc then to add into another project B?
Yes; that "project A" "bundles" a lot of my custom packages (as well as various other commonly used packages) as subdependencies, for consistent versioning of them all together -- but with yalc in-use so that I can swiftly apply local modifications to any of those subdependencies, through that "bundling package" (ie. "package A" in your text).
Yes, that is a good idea; I'll try to get to it eventually. (I'm backlogged in development tasks atm, and my "zalc" wrapper currently gets the job done, but I'll keep it in mind.)
For anybody who might be stuck with this "nested dependencies" issue, this worked for me.
(this solution is already mentioned way up here, but just in case)
For each dependency that has nested dependencies, add .yalc to list of files array in package json. like
"files": [
"README.md",
"dist",
"src",
"tsconfig.json",
...
".yalc"
],
and run yalc publish --push or yalc push from this package. This will push the dependency package with '.yalc' directory included in it, hence fixing this error.