maxhoesel-ansible/ansible-collection-smallstep

About molecule-docker+systemd issues and migrating Molecule tests to Podman...

maxhoesel opened this issue · 2 comments

maxhoesel commented

As outlined here and here, newer releases of systemd don't play nice with systemd docker containers. This causes a "no such file or directory" error when running our molecule scenarios on newer distros (Fedora 34+/Archlinux).

For now, a simple workaround is to set the systemd.unified_cgroup_hierarchy kernel parameter to 0 on boot to force usage of cgroupv1.

It should be noted that the CI is currently unaffected, as it is still running on Ubuntu 20.04.
That said, we should probably look into a better long-term solution if the "systemd in docker" hack really is going away:

  • Option A: Use VMs, probably via Vagrant. I don't like this option because it would mean massively increased resource usage and test runtimes
  • Option B: Migrate a container runtime that supports systemd containers like Podman. This could be a good approach for the CI, but I'd like to preserve an option for running tests locally through the CT runtime that most of us already have installed - Docker.
  • Option C: Do nothing and hope that we get a new option for running systemd in docker. This would be my preferred option. Thankfully, this issue is not very urgent, so that's probably what I'll do for now.
maxhoesel commented

Update: I've been working on migrating the collection over to podman for molecule tests, but this is still in an early stage. Here's a list of issues that i've encountered:

  • We need a container network with DNS resolving capabilities for some of our molecule tests. To do this, you need to install a separate dnsname plugin in addition to podman, and the ubuntu package for that plugin is currently broken (see the CONTRIBUTING.md on the feature branch).
  • Some of the CI containers fail with a "Failed to create temporary directory" message and I have no idea why.
  • When i run the tests on my machine I get inotify errors, informing me that I've hit a limit.
  • Rootful containers (which would probably resolve a fair share of these issues) are not supported by molecule yet.

There's also some smaller issues - notably, the podman driver doesn't clear up the created molecule_smallstep network, but that's not a big deal.

You can check out the podman branch if you'd like to take a closer look

maxhoesel commented

Going to give this another shot this weekend. Given that systemd plans to remove support for cgroupv1 in a future release, moving to a container runtime that supports systemd only makes sense