Constraint syntax is not expressive enough.
dcoutts opened this issue · 39 comments
Here is a property that we would quite like the solver to have:
given any solver result/solution we should be able to construct a set of solver input constraints that uniquely determine the same result (in any environment in which that solution exists)
Or to put it another way, we should be able to freeze the result of the solver and feed it back into the solver later to get the same result. The solver almost has this property but not quite. In particular the introduction of setup deps and qualified goals means that simple "pkg == version" constraints are no longer expressive enough to describe all the solutions the solver can produce. If the solver picks two versions of package foo (e.g. one a "main" lib package and another as a setup dep) then we cannot just constrain "foo == 1.0", we would need to specify the scope of the constraint. Notice also that "primary" vs "setup" dep scope is not enough, since we could pick two versions of foo both as setup deps. Probably what is required to express all the solutions is something corresponding to the internal goal qualifications.
Does freeze need to handle changes in the set of installed packages?
@grayjay I'm not sure I totally understand the question. Want to grab me on IRC in #hackage for a discussion? Or ask again with more details.
The new freeze command (in PR #3503) takes the current solution (having made sure that was up to date) and translates it into a set of constraints and then writes it out. The new build/repl/configure/freeze all will re-run the solver if the solver inputs would change, so this includes changes in the set of installed packages in the global package db (but not in the user or store dbs) and changes in the set of source pakages from hackage. In the new scheme we never tell the solver about installed packages in the store, just source packages and the global installed packages. I'm not sure if that answers the question.
@dcoutts I was wondering what should happen when a user runs new-freeze, removes a globally installed package, and then runs new-configure again. Should new-freeze just freeze the version numbers, so that cabal can try to rebuild the previously installed package? Or does it need to ensure that the dependencies are exactly the same, including the flag assignments? I joined #hackage, if you want to discuss it there.
Ah ok. So if at the point when we ran new-freeze the solution involved an installed package from the global db, (e.g. the copy of containers that is bundled with ghc) then yes we only know the version and not flags, so we can only generate version constraints, not flag constraints. For all packages outside of the global db we do know all the flags and do produce flag constraints with new-freeze.
So you're right to point out that this is a slight hole, however it's not one I'm really worried about. While at the moment we use the whole global package db, our assumption is that these days it only really includes a small set. We can make that assumption properly true by using a "core" package db that really only contains the core packages. Also, of the existing packages typically in the global db, I'm not aware of any with automatic flags that significantly affect things.
@dcoutts Thanks for explaining. I'm still not very familiar with the overall design for new-build, but I think that using goal qualifiers in freeze constraints makes sense. The only problem that I can think of is that we might add flags to modify goal qualification, such as --independent-goals. It's probably not a big deal, though, since we could just specify the goal qualification scheme in the freeze file.
@dcoutts In principle, I agree. But for the use case in question, wouldn't it be even better if we'd have a dedicated new solver (we could call it --explicit-solver or --trivial-solver) that won't solve at all, but simply deserialize an install plan that is passed via some mechanism. The sanity check would still run, of course.
Or are there particular reasons why you want to have this feature in connection with being able to make some modifications, and need the actual solver machinery in the background?
We actually already have something like --trivial-solver in Cabal, it's called --explicit-configuration. Perhaps we only need to port this feature to cabal-install?
@dcoutts Ok, I get it. The only thing that worries me is that I still don't consider the whole setup dependency / qualified goals stuff to be extremely stable. If we allow constraint syntax for these kinds of constraints, we may expose rather fragile solver internals to the end user.
So, syntax proposals welcome.
I need this feature for other reasons; specifically, when I build cabal-install's test suite from source with cabal-install HEAD, I accidentally pick the in-tree Cabal library to build the custom setup for happy with (which causes it to be built inplace). It would be far better if I could specify a constraint that the custom setup should NOT use the inplace Cabal. Syntax here (along with a way to distinguish between local and remote) would go a long way.
In #3932 we wanted to add a constraint that all versions of Cabal which are brought in from setup dependencies must be >= 1.20. Since we don't have qualified constraints, the best we can do is just say that Cabal, ANYWHERE, must have >= 1.20. That's suboptimal; a more precise constraint would be better. Once we fix this, we can make the constraint in @dagit's patch more precise.
A constraint that applies to all packages' setup scripts doesn't correspond to a qualifier in the solver, but it probably wouldn't be much harder to implement.
@robjhen Are you still interested in working on this issue?
@grayjay Yes I'd be interested but I've been trying with no success to get the latest Haskell Platform and hence the Cabal project to build on my current operating system, Debian 7.
I've uploaded the code from the hackathon: 2d9c460
To install Debian 8 and then complete this feature would likely take me about two months as I have to fit it into limited spare time. I guess that may be too long to wait? Let me know if either a) you would prefer that someone else who has a working build environment take over development or b) you don't mind it taking that amount of time. Sorry about this!
I don't think it's a blocker for 2.0. Feels important though; I'd put it on the default list (indeed I just have.)
Thanks. So two months would be soon enough?
Yes, if software development time estimates ever meant anything ;)
Okay, great! I'll go ahead with the work, and get back to you.
Just to let you know I'm close to finishing this. I'm currently splitting my changes up into separate commits for ease of review and I should have a pull request ready by tomorrow.
Awesome, looking forward to it.
I've made pull request #4211 - this is some refactoring prior to my main submission.
Can this be closed, seeing as #4219 was merged?
See PR #4236
I see what you did there :). I'm probably not qualified, but I'll see if I get time to review that PR tomorrow.
If that concludes the changes you should probably just add a "Fixes: #4236" to one of the commit messages. That'll auto-close this when it gets merged :).
@BardurArantsson I have another PR pending review (see previous comment), and then I believe @grayjay will want to upgrade the solver to act on the the new qualified constraints before we close this issue.
Oh, ok, fair enough. :)
Sorry, didn't refresh my browser window there so I missed your last comment before making mine :) Anyway, I would check with @grayjay before closing the issue as she may or may not want to leave it open.
Yes, this issue isn't finished, because the solver doesn't use the constraint qualifiers yet. Another thing that needs to be done is printing the qualified constraints in the freeze file.
cabal now has qualified constraints that apply to setup dependencies, top-level dependencies, and any dependency. We'll probably need to implement per-component dependency solving before we add qualified constraints for build tools.
Related pull requests: #4211, #4219, #4228, #4236, #4252, #4257, #4342
We should be able to add setup qualified constraints to freeze files now. We can continue to constrain build tool dependencies with unqualified constraints that allow multiple versions.
Should this ticket also cover Hackage revision constraints, or should there be a different one? There's also the idea of allowing index-state in freeze files.
I think both of those features could be implemented separately from qualified constraints, so they should probably be separate issues.
What's actually left to do for this issue? Qualified constraints seem to just work now.
v2-freeze is still unaware of that, but there are other issues filed for that.
Ah - executable constraints. foo:bar:exe.baz == 1 will be generated, but not parsed.
If nobody takes ownership (i.e. assigns to themselves) before Thursday 2019-12-05, I'll move this issue to 3.4 milestone.
remilestoning for 3.4
I had to deprecate the latest happy release because of this :(. there is no way to put in a cabal.project file a flags: .... that should apply to qualified goals, like happy used as a build-tool-depends.