UI for freezing dependencies

Question

UI for freezing dependencies

gregwebs opened this issue 11 years ago · 26 comments

see #1499 for understanding dependency freezing and asking fundamental questions about this process.

First build of my application:

cabal install --only-dependencies --freeze-dependencies  # generates a cabal.config
git add cabal.config
git commit -m "Added cabal.config"

Another user pulls the git repo and installs the frozen dependencies:

cabal install --only-dependencies

Need to update dependency:

Change e.g. foo == 1.0.* to foo == 1.1.* in .cabal
cabal install --only-dependencies to perform a conservative upgrade which may change more than just foo
This step can't work right now since the constraints field is designed for users to modify
git status

modified: cabal.config
modified: project.cabal

git diff to view all dependencies of dependencies changed in cabal.config
test out new dependencies locally
git commit -am "updated foo dependency"

The UI change above is the addition of --freeze-dependencies.

The problem is that constraints: in cabal.config was designed for users to modify, not for cabal to re-write (and over-write what a user changed). We need either a new frozen-dependencies: field that the user is not supposed to modify, or an entirely new file that lists frozen dependencies.

It is important to note that we need per-component freezing (even if it does not work properly today), so a frozen dependencies: field would look different than the current constraints: field.

tibbe commented 10 years ago

Thanks!

Answer 1 · 2013-09-11T18:36:09.000Z

How about this as a first easy step: Add --freeze-dependencies and '--overwrite-configtocabal install`. That way you can use both flags when you update your dependency on e.g. foo and then manually just revert any cabal.config updates cabal made that you didn't like.

Answer 2 · 2013-09-11T19:15:58.000Z

That sounds good, but is --overwite-config needed? It seems I could just type --freeze-dependencies again. I think the initial --freeze-dependencies would need to overwrite the config if it existed anyways.

Answer 3 · 2013-09-11T20:00:38.000Z

You're right. We don't need --overwite-config. --freeze-dependencies is enough and should just edit the cabal.config file (i.e. read, change constraints, write).

Answer 4 · 2013-09-11T21:39:14.000Z

what about dealing with per-component issues?

I am fine with just doing a global constraint right now, basically just using the cabal-constraints code, just tell me where to put it.

https://github.com/benarmston/cabal-constraints

Answer 5 · 2013-09-12T01:14:12.000Z

I'm pleased that you like the cabal-constraints code. I've tidied it up a little and pushed in the last few minutes. If you've already taken it you may wish to see if you want those changes too.

Answer 6 · 2013-09-12T03:09:57.000Z

doh, I just saw this: https://github.com/wereHamster/cabal-lock/blob/master/Setup.hs
@benarmston do you want to send in a patch to the cabal project with your code? Either way I will help see freezing to fruition.

Answer 7 · 2013-09-12T08:25:20.000Z

@gregwebs I'd be happy to send in a patch. A quick look at https://github.com/wereHamster/cabal-lock/blob/master/Setup.hs suggests that I should find where confHook is called and look for a suitable place to add the code from there.

I should be able to find time for this over the next couple of days.

Answer 8 · 2013-09-12T16:41:36.000Z

I spoke with @dcoutts a bit. We cannot do the per component thing anytime soon. configure requires that all the components in the package have compatible dependencies. In addition, the constraints syntax doesn't support it either.

My suggestion is that we -- and by we I mean someone else who has time -- do the simple thing first: add a --freeze-dependencies flag to install and have it write/edit cabal.config.

Answer 9 · 2013-09-12T16:59:07.000Z

Have we considered making it a new top level command, rather than abusing install?

Or if we end up with lots of these kinds of editing commands, group them under a common command.

Answer 10 · 2013-09-12T17:35:42.000Z

@dcoutts I'd be fine with that too. It's really only half an install anyway (i.e. the dep resolve part, not the install part).

Answer 11 · 2013-09-12T19:11:13.000Z

This sounds like a good first implementation, I don't want the perfect to be the enemy of the good.

Answer 12 · 2013-09-13T18:43:58.000Z

I've made a little progress on this, which can be seen here.

Briefly; during the configuration action, once the LocalBuildInfo is available, it is used to determine the exact constraints. The cabal.config file is then unconditionally overwritten with all of those constraints.

@tibbe you mentioned that this was related to the dependency resolving part of the install. Does that mean that we'd be better off with this functionality Distribution.Client.Install? Perhaps making use of InstallPlan? If I've understood your comment correctly, would you mind explaining why LocalBuildInfo isn't sufficient to freeze the dependencies?

Answer 13 · 2013-09-13T18:55:04.000Z

I think LocalBuildInfo is the wrong phase and place.

We're talking about cabal-install here, so we have the InstallPlan. So we can just inspect the InstallPlan and we get all the info we need about which versions we are picking.

In principle it's wrong to get it from the LocalBuildInfo because such a thing need not even exist. That is a detail of the "Simple" build system which may or may not be in use by the package in question. Also if you're using LocalBuildInfo then it means you're in the Cabal lib and there you're not allowed to read/write cabal.config files, because those belong to the package tool, not the build system. Freezing is a feature of the package tool (cabal-install) not the build system (Cabal).

Answer 14 · 2013-09-13T19:41:03.000Z

@dcoutts Is it possible to extract flag constraints from the install plan?

Answer 15 · 2013-09-13T21:13:56.000Z

@dcoutts thanks for your response and explanation. I've updated my branch with what I hope is something more appropriate.

This time InstallPlan is being used by processInstallPlan to unconditionally overwrite cabal.config. Am I closer to what you would expect this time?

Answer 16 · 2013-09-13T21:22:10.000Z

@benarmston Could we instead read the old file, update the constraints, and write it out again? Right now you're losing any other settings the users might have put in the file (e.g. library-profiling: False). It's OK if we cannot preserve formatting and comments, but not losing the settings would be nice.

Answer 17 · 2013-09-13T21:44:55.000Z

@tibbe of course! I'd prefer to leave such work until I'm approaching this in the right manner though. Without any previous familiarity with Cabal's or cabal-install's implementation, it's hard for me to judge that myself. Hence my presentation of what I'd got so far.

I figure that the following all need addressing:

Make the overwriting conditional.
Perhaps omit certain dependencies, such as base and the package itself.
Replace only the constraints: section of cabal.config.

And probably other things too. Thanks for the feed back.

Answer 18 · 2013-09-16T11:14:09.000Z

I still think this should be a separate command, and internally not intermingled with the complex process of actually installing packages. It's fine for another top-level command to run the solver etc. Take a look at the fetch command for example. Once it has the InstallPlan it can do whatever is needed, without having to mingle it with the complexities of processInstallPlan etc.

Answer 19 · 2013-09-21T22:32:30.000Z

I have made some progress on implementing this as a separate top-level command, which can be seen here

The new command , freeze, takes a number of options related to resolving dependencies,
namely, --solver, --max-backjumps, reorder-goals and --shadow-installed-packages.

I'd like to get some feedback on which additional options, if any, should be available. In particular the options --flags, --enable-{tests,benchmarks}, --constraint and --preference could all affect the dependencies or their version resolution. So all seem like they should be available. Is this correct?

Also, does the duplication of so many options with install not bother anyone?

The options concerned with compilers may prove more difficult.
Different versions of GHC ship with different library versions, the
constraints that are written currently include all dependencies, including
base. This prevents the constraints, as written, from being used with
alternate versions of GHC.

There are three solutions to this problem: 1) have the user edit the
constraints section, 2) exclude certain packages from the list of constraints,
3) develop a mechanism for per-arch-os-compiler constraining. As neither (2)
nor (3) have been developed we default to (1).

The lack of a good story for per-compiler constraints has lead to the options
--with-compiler, --ghc, --uhc et al, not being supported. My suggestion is that these can be revisited at a later date if and when they are needed.

There are still a number of issues left to address, including:

The cabal.config file path is hardcoded.
The cabal.config file is completely overwritten. Just the constraints section should be overwritten.

Any feedback anyone has on this would be very much appreciated.

Answer 20 · 2013-09-22T04:56:54.000Z

I don't see it as a big problem that compiler versions can't be changed. A compiler can be considered part of the freeze. What compiles with one version of GHC doesn't necessarily compile on the next. For industry use everyone should have the same compiler version.

The problem would be for the use case of distributing an open source project executable. But freezing might not be such a good idea though because again the frozen deps are only good for the compiler they were froze on. The true way to freeze would be to distribute a binary.

Answer 21 · 2013-09-23T22:40:51.000Z

OK. Then we are agreed that there is no need, perhaps even sense, in trying to support per-compiler freezing. At least at the moment.

You're comment about distributing open source projects with constrained dependency versions is interesting. My own use-case for wanting this is to be able to have repeatable builds on Heroku, which requires that I commit the cabal.config file to git. If I were to then make my project available, I would be over constraining the dependencies for other developers. Perhaps the solution to this is a more sophisticated build / release process.

Answer 22 · 2013-09-23T23:06:27.000Z

AFAIK Ruby gems supports different compiler dependencies in the Gemfile (.cabal file) but you only lock it down to one compiler for your application and isn't designed to freeze to different compilers/interpreters for distributing an open source executable. I think it can somehow remember what was locked down in different environments, but that is still for application deployment, not distribution.

Answer 23 · 2013-09-23T23:16:53.000Z

For your last 2 points: most people aren't using cabal.config now, so we have a minimum viable freezing implementation now. I feel like there is going to be a bunch of wasted work to solve them with the current design. Both points would be solved by putting this output into a file cabal.freeze that is designed to never be touched by a human. That will make it easier to have per-component or per-compiler dependencies in the future, and lets us add other meta-data as needed. The file location could be special (project root) or it could be necessary to list it in cabal.config.

Answer 24 · 2013-09-24T21:57:04.000Z

@gregwebs in the Ruby world, Gemfile and Gemfile.lock are managed by a tool called Bundler which in turn makes use of Rubygems.

The Gemfile syntax allows for a lot of interesting things. It's certainly possible to specify dependencies for a particular compiler or platform. Such dependencies are always included in the dependency resolution but may not be installed depending on the platform. I think that the platform-specific dependencies are still written to Gemfile.lock but I'm not certain.

At work we control our platform-specific gems by always installing them and then controlling whether they are included into the running application or not. IIRC, this was to allow the downloaded gems to be committed into git without suffering commit wars between different developers on different platforms. Committing the gems means that our build system doesn't have to download them again, which can be a real pain given that Bundler is not the fastest tool in the box, insulates from temporary network failures and...

I'm not that involved in the open source Ruby community but I believe that it is very common to have Gemfile.lock committed to git and all developers use the same set of dependencies. This is more manageable given that the Ruby community is much more inclined to use the latest-hotness and a little less inclined to worry about backwards compatibility than the Haskell community. At least that's how I see it.

From my experience of using Bundler, I think it makes a good model to base dependency freezing on. As you mentioned before, the Gemfile.lock format of specifying where all dependencies come from can be very useful. So I'd agree that we should be heading in the direction of a cabal.lock/cabal.freeze file containing a similar set of information.

Answer 25 · 2014-04-05T22:39:22.000Z

I believe that this can be closed now.