Snapshot curator tool for, e.g., creating Stackage snapshots.
This is the "curator 2.0", replacing https://github.com/fpco/stackage-curator. It relies on pantry for finding appropriate packages, and Stack for performing the builds. It is intended to be much simpler to maintain than the old stackage-curator tool.
This tool is not yet complete. Here's a (likely incomplete) list of
things that still need to be handled to replace stackage-curator:
- Collect the Haddocks in a way that stackage-server can handle them
- Proper CLI, right now the
app/Main.hsjust runs through a bunch of steps. We need to have individual commands like the current tool, so each command can be called in an appropriately locked-down Docker container. - Logic for uploading generated snapshots and other info to Github, S3, etc.
- Ability to roll an LTS minor version bump.
- Ability to specify package locations from Git.
- External, but: stackage-server needs to be updated to support the new snapshot format/location
- No support for custom configure arguments from
build-constraints.yaml. I'd like to see if we can get rid of them entirely and instead just customize the Docker build image.
Here's a rundown of how this tool is intended to be used.
We update the Hackage index to get a list of all of the most recent
package versions. This is pantry's updateHackageIndex command.
We start with build-constraints.yaml, the configuration file in
commercialhaskell/stackage. This specifies all of the packages we want
to include in a snapshot, along with a bunch of configuration.
We parse build-constraints.yaml and convert it into the
constraints.yaml file, which contains a more properly structures set
of constraints. We'll continue to let users edit the
build-constraints.yaml file, since it's more user-friendly. But
constraints.yaml gives us more flexibility.
- For LTS minor bumps, instead of generating
constraints.yamlfrombuild-constraints.yaml, we'll take theconstraints.yamlused for the last LTS release in the series. Details still need to be worked out on how upper bounds are added and where this file is stored.
Curator team: at this point, you can edit constraints.yaml to make
tweaks to the build plan. This replaces the old CONSTRAINTS
environment variable.
We combine the constraints.yaml file and the information from
Hackage to produce snapshot-incomplete.yaml. This has a concrete
list of all of the packages we intend to include in the
snapshot. Again, this file can be manually modified if desired.
- When we support Git repos, we'll also be checking those repos to find the latest appropriate release. We'll need to figure out exactly how that plays in with LTS upper bounds; I'm thinking we'll have logic like "use commit X, or the latest if it meets version range Y."
The snapshot-incomplete.yaml file does not have all of the
cryptographic hashes necessary for fully reproducible builds. We next
generate snapshot.yaml with all of this information. This file
should never be manually edited, instead edits should occur at the
snapshot-incomplete.yaml and constraints.yaml phases.
The snapshot.yaml file gets checked for its consistency ensuring the
following:
-
All package dependencies are explicitly specified in constraints files
-
Dependency bounds of all snapshot packages are properly satisfied
-
There are no dependency cycles in the snapshot (only libraries and executables are included into checked dependency tree as test suites and benchmarks are allowed to create cycles)
-
The snapshot contains Cabal version suitable to build all packages
We unpack all of the package specified by snapshot.yaml into a local
directory, and generate a stack.yaml that gives instructions to
build all of those packages.
We build the packages, run test suites, and generate Haddocks.
TODO Grab artifacts and upload them to the right place.