bazel-contrib/rules_oci

Proper support for installing packages with apt

alexeagle opened this issue · 14 comments

alexeagle commented

https://github.com/chainguard-dev/rules_apko provides a nice solution for Alpine. We may want something similar for Debian.

apt-get does things like:

We would like to read a lockfile telling us where to download the .deb files, but apt-get doesn't create one. Something like https://github.com/TrevorSundberg/apt-lock looks promising, if we could rely on it.

alexeagle commented

Ooh! What's your plan with long term maintenance and support of that repo?

betaboon commented

i have no explicit plans. it's working for us as is.

I'm fine with working on it to help this case.
maybe forking it as a basis for a rules_apto :P
or donating it.
whatever helps. let me know ;)

alexeagle commented

Nice, it looks like a pretty good shape to me. Maybe we can donate that ruleset to bazel-contrib and get it on the BCR. @thesayyn any thoughts on it?

thesayyn commented

Look promising. This is what distroless team does. I have done something similar in the past; https://github.com/bazel-contrib/rules_oci/tree/apt/experimental/apt, a pure bazel replacement using repository rules. The problem with apt packages is that they are rolling so it's impossible to get reproducible builds even if you generate locks etc, the remote .deb may disappear when there is a new version.

thesayyn commented

debian snapshot is what we want but they pretty much unreliable.

betaboon commented

snapshots is what I'm using in rules_debian_packages

alexeagle commented

I think users would need to "patch over" this problem with a reliable, immutable registry. Something like PackageCloud, Artifactory, etc. There's already a need for such a thing to back the Bazel downloader, and I think we can provide this to Aspect Pro users and everyone else will have to build their own.

Which is to say, users of "bazel-contrib/rules_debian_pkg" or whatever we call it will be required to supply the package registry they want to use, and choosing snapshot.debian.org is one choice they could make but we warn them that it's non-reproducible.

betaboon commented

tbh i found debian snapshots to be quite reliable in terms of old packages being available.
the one aspect that snapshots.debian.org is unreliable at is the occasional timeout.
so i would heavily advice a caching proxy like squid et al

also, the lockfile-format is heavily inspired by the lockfile that distroless uses. (it might even be compatible)

thesayyn commented

that it's non-reproducible.

problem with debian snapshot is that it's unreliable in terms of uptime. you will get frequent interruptions because the load that registry is enormous.

  • 👍1
betaboon commented

just a quick update, I'm currently working on making rules_debian_packages usable with rules_oci.

i got this working on a dev-branch:

load("@apt//:packages.bzl", "debian_package_layer")
load("@rules_oci//oci:defs.bzl", "oci_image", "oci_tarball")
load("@rules_pkg//pkg:mappings.bzl", "pkg_mklink")
load("@rules_pkg//pkg:tar.bzl", "pkg_tar")

pkg_mklink(
    name = "usr_bin_bash",
    link_name = "/usr/bin/bash",
    target = "/bin/bash",
)

pkg_tar(
    name = "usr_bin_bash_layer",
    srcs = [":usr_bin_bash"],
)

oci_image(
    name = "image",
    base = "@distroless_cc",
    tars = [
        # required to replicate distroless/python
        debian_package_layer("libbz2-1.0"),
        debian_package_layer("libcom-err2"),
        debian_package_layer("libcrypt1"),
        debian_package_layer("libexpat1"),
        debian_package_layer("liblzma5"),
        debian_package_layer("libreadline8"),
        debian_package_layer("zlib1g"),
        # required by rules_py
        debian_package_layer("bash"),
        debian_package_layer("coreutils"),
        debian_package_layer("grep"),
        ":usr_bin_bash_layer",
    ],
)

oci_tarball(
    name = "image.tar",
    image = ":image",
    repo_tags = ["python_base:latest"],
)
  • 🎉1
  • ❤️7
  • 👀2
betaboon commented

just wanted to let you know that i just merged the rules_oci support in rules_debian_packages mentioned above.

I'm still missing bzlmod support, and could use some some help on that, as i have no experience with bzlmod yet (see wip PR )

alexeagle commented

That's neat! We are working with the distroless team who already has something similar for working with debian packages, I'll leave for @thesayyn to propose how https://github.com/GoogleContainerTools/rules_distroless might relate to rules_debian_packages.