Ensuring everything is caught by the subpackages

Question

Ensuring everything is caught by the subpackages

Closed this issue 6 years ago · 10 comments

When packaging multiple subpackages with conda, it is possible to "forget" to include files that would have otherwise been installed.

When packaging (modern, I can't really speak for early versions of RPM packaging) the packaging software will simply FAIL letting you know that you forgot to include certain files. that would have otherwise been installed with the "catchall"

Would it be nice if conda did that too?

As an addendum:
I also noticed that when it comes time to selecting files, the prefix is littered with conda stuff, that make it hard to for example:

blanket install everything in bin in the -exe package for example.

Answer 1 · 2018-08-21T07:59:38.000Z

Conda is aware of "conda stuff" in $PREFIX (or more correctly, the necessary dependencies of the package you are building and installing to that prefix). This is essential to how conda-build operates.

However you can ignore all of those, conda build knows about them and filters them out for you.

Answer 2 · 2018-08-21T12:24:54.000Z

@mingwandroid thanks for making me try again. I swear it was including them in the package. But maybe I was just seeing things.

Answer 3 · 2018-08-21T12:31:24.000Z

@msarahan, your fact checking would be appreciated.

For the catchall suggestion it may make some sense but it needs more discussion.

One problem you'll find is that the sub-package for which you'll most likely want to use catchall will tend to be a dependency of the other sub-packages so it will get built first, and at that point, the consideration of 'all' will include bits (it'd include everything) you'd prefer to have ended up in one of the dependencies.

I suppose catchall could be interpreted as the remainder after removing the output/files from the other packages though? A complication with this is that you can have outputs that use a script to install the files instead so there's no output/files for those.

Answer 4 · 2018-08-21T13:01:05.000Z

conda-build does know about them, but only ignores them when they're not specifically requested to be included. Adding entries manually, as suggested in #3090 is a specific request to include them.

The notion of a catch-all is a large increase in the knowledge that conda-build must have about all the packages going into a build. It also implies some kind of ordering among outputs that currently does not exist. @hmaarrfk if you want this feature, please consider a PR. We will not implement this ourselves, especially since we almost exclusively use scripts for installing files in prefixes, rather than the files listing.

Answer 5 · 2018-08-21T13:06:49.000Z

@msarahan understood. is there a way to "uninstall" dependencies before specifying which files are included?

Answer 6 · 2018-08-21T13:29:17.000Z

You seem to be hitting your head on a wall with the files way of specifying subpackages. Is there any reason why you can't write a script that copies your files from the workdir into the $PREFIX? That will exclude things that are already there from other dependencies. The files way of doing things is much more simplistic and explicit, which really limits it to very simple things.

Answer 7 · 2018-08-21T13:53:36.000Z

I guess I tried to split the build step into "compile all" and "make install", then remove what I don't want, but for some reason that didn't work out.

Answer 8 · 2018-08-21T14:06:12.000Z

Hmmm... you might be cramming too much into the build step. We have found it best to have only "compile all" in the build step, then do install stuff in the output scripts. GCC is perhaps the most complicated example of this, and it has taken many iterations to get right: https://github.com/AnacondaRecipes/aggregate/tree/master/ctng-compilers-feedstock/recipe

Boost is a little bit simpler: https://github.com/AnacondaRecipes/boost-feedstock/tree/master/recipe

As you can see from these, there's no magic. Most software is not written with the assumption that the pieces can be installed separately. Most makefiles don't do a very good job of separating concerns completely. You often have to do a make install sort of step, and then follow it up with copying more files, or removing files that you don't want.

I don't believe there is a general fix to conda-build for this. If anything, we all should be submitting PRs upstream to help their install scripts work better.

Answer 9 · 2018-08-29T15:52:42.000Z

Thanks @msarahan I think I had a small misunderstsanding about how files installed during the top level build step were treated.

Answer 10 · 2022-03-14T06:04:32.000Z

Hi there, thank you for your contribution!

This issue has been automatically locked because it has not had recent activity after being closed.

Please open a new issue if needed.

Thanks!