tazjin/nixery

Consider support building nix-snapshotter images

elpdt852 opened this issue · 4 comments

elpdt852 commented

Hi there, thanks for the awesome project! We just open sourced nix-snapshotter which brings native understanding of Nix packages to containerd, i.e. the image manifests become just metadata and we use the Nix protocols to pull the packages and bind mount them into the container rootfs.

We're also written in Golang, so I imagine it won't be too hard to hook up our public methods to generate nix-snapshotter images based on API requests. Note that although nix-snapshotter is backwards compatible with non-Nix and existing Nix images, Docker/containerd not configured with nix-snapshotter is unable to run nix-snapshotter images. So this probably needs to be behind a feature flag.

We also have a FAQ entry also to help explain the differences between nix-snapshotter and nixery.

tazjin commented

Your project is GREAT! Been waiting for something like this to come along for a while!

Yes, totally in favour. Can we discuss what would need to be done on some chat medium (IRC, Matrix for the official TVL chats, documented on https://tvl.fyi), or maybe in a call (I'm in UTC+3, and fairly available)?

By the way, there's a mistake in your FAQ entry:

but still uses the same layering strategy as upstream's pkgs.dockerTools.buildImage (see above)

Nixery does not actually do that. It uses a more sophisticated mechanism: https://tazj.in/blog/nixery-layers

However, of course, the whole problem doesn't exist if you don't have the Docker layer restriction :)

tazjin commented

Notes from the catchup with @elpdt852 just now:

  • in one of its modes, nix-snapshotter essentially uses a standard OCI manifest with a single layer containing empty folders (as mountpoints) for the various store paths to be overlaid
  • the store paths to include are specified in annotations on the layer inside the manifest
  • those layers can be mixed arbitrarily with non-nix-snapshotter layers (i.e. standard OCI images)
  • integration with Nixery could be done by having Nixery serve this manifest and the layer with the right annotations
  • Nixery doesn't have to care (from its side) about using it together with standard OCI layers
  • code to do some of this already exists in nix-snapshotter, and Nixery can call it directly, for example:
    https://github.com/pdtpartners/nix-snapshotter/blob/main/pkg/nix2container/generate.go#L50-L52
  • Nixery would not serve the store paths itself in this case: it would be necessary for the client's Nix installation (if the default nix-based "layer fetcher" is used) to have the right cache configured
  • for nixery.dev, people would have to configure the TVL cache and we'd have to use a special image prefix for these images, we'll see
  • for layer-fetching, it might be interesting to look at integration with tvix-store. I'm not up-to-date on the state there, so I deferred this question to @flokli
  • 🚀2
tazjin commented

Reopening, that PR above is only related to this issue, it doesn't close it.

elpdt852 commented

Sorry, looks like that was automatic!