use distro python module instead of lsb_release binary

Question

use distro python module instead of lsb_release binary

FooBarQuaxx opened this issue 6 years ago · 5 comments

I've tried to use apt-mirror-updater from a vanilla Ubuntu 18.04 docker image to set the fastest apt mirrors. I've encountered an error because python-executor relies on lsb_release binary to fetch the distribution codename and id. Here's the exception traceback:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/apt_mirror_updater/cli.py", line 160, in main
    callback()
  File "/usr/local/lib/python2.7/dist-packages/apt_mirror_updater/__init__.py", line 425, in change_mirror
    new_mirror = self.best_mirror
  File "/usr/local/lib/python2.7/dist-packages/property_manager/__init__.py", line 781, in __get__
    value = super(custom_property, self).__get__(obj, type)
  File "/usr/local/lib/python2.7/dist-packages/apt_mirror_updater/__init__.py", line 176, in best_mirror
    if self.release_is_eol:
  File "/usr/local/lib/python2.7/dist-packages/property_manager/__init__.py", line 781, in __get__
    value = super(custom_property, self).__get__(obj, type)
  File "/usr/local/lib/python2.7/dist-packages/apt_mirror_updater/__init__.py", line 343, in release_is_eol
    if hasattr(self.backend, 'get_eol_date'):
  File "/usr/local/lib/python2.7/dist-packages/property_manager/__init__.py", line 781, in __get__
    value = super(custom_property, self).__get__(obj, type)
  File "/usr/local/lib/python2.7/dist-packages/apt_mirror_updater/__init__.py", line 164, in backend
    return sys.modules[module_path]
  File "/usr/local/lib/python2.7/dist-packages/apt_mirror_updater/__init__.py", line 164, in backend
    return sys.modules[module_path]

I've debugged the value of self.distribution_id and self.distribution_codename and both are empty.

Since there's no obvious way to enforce the existence of lsb_release inside a python module definition, I suggest using python distro module to get that information.

Answer 1 · 2018-10-07T11:09:51.000Z

Hi and thanks for the feedback.

I've run into this problem myself when setting up chroots because debootstrap doesn't install the lsb-release package which means the lsb_release program is unavailable in a fresh chroot, breaking executor support. In fact this is the reason why the create_chroot() method in apt-mirror-updater takes the liberty of installing the lsb-release package after creating a new chroot 😊.

Why the distro module isn't an option: Switching to the platform.linux_distribution() function (or its replacement in the distro package) doesn't provide a full solution because executor may not be running on the system where it's executing commands, in fact Python might not even be installed in the target environment! (which may be the local system or a chroot or a remote system available over SSH)

I appreciate that most likely none of this flexibility means anything to your specific use case, but nevertheless I've intended for executor to work (the same) in all of these target environments.

Previous investigations: I've investigated independence from the lsb_release program before, but that investigation seemed to open up Pandora's box, which stalled actual improvements. The main thing is that lsb_release provides a nice and simple abstraction and removing that amounts to re-implementing it, which I'm not really looking forward to 😇 (it gets really nasty, having to parse files like /etc/apt/sources.list)

What now? While I will never be able to make this fool proof in every circumstance imaginable, I do know that I can improve on the status quo and when I do I will follow up here.

Answer 2 · 2018-10-07T11:32:31.000Z

Since I'm looking into this by gathering intel right now, but I already know that it's going to get hairy (complex) based on previous investigation, I decided to document my findings here until such a time that I feel like I understand things well enough to improve the executor package.

Investigation based on a Debian testing chroot:

Last night I created a minimal Debian testing chroot to experiment with and found the following:

As expected the minimal install does not contain the lsb-release package nor the lsb_release program nor the /etc/lsb-release datafile.
The file /etc/os-release exists but the contents are a bit disappointing, because the only useful bit of information (buster/sid) is embedded inside the PRETTY_NAME variable, not in a separate variable:

PRETTY_NAME="Debian GNU/Linux buster/sid"
NAME="Debian GNU/Linux"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

The file /etc/debian_version exists and contains the string buster/sid which is good because it means I won't have to deal with the awkward encoding of this information in /etc/os-release.

Some further research for my own insight on what this buster/sid text actually means:

buster is the next generation Debian release, which makes sense because I created a chroot from the "symbolic" testing release.
sid is the name of the unstable release: "While other release code names progress in time from being testing to being stable, Sid is forever doomed to being unstable."

Practically speaking I should now be able to implement a fall back code path in distribution_codename for Debian (but not Ubuntu ¹) systems that activates when the lsb_release program isn't installed, it would work by splitting the contents of /etc/debian_version on / and taking the first component.

¹ On Ubuntu systems /etc/debian_version matches the contents found on Debian systems, that is to say it describes the Debian release from which a given installation derives and not the Ubuntu release.

Answer 3 · 2018-10-07T12:28:50.000Z

Investigation based on an Ubuntu 18.04 (bionic) chroot:

To compare and contrast with my experiences gained in a Debian testing chroot I also created an Ubuntu 18.04 chroot using debootstrap and to my surprise the lsb-release package and lsb_release program are installed out of the box!

I triple checked to make sure I wasn't just confusing myself but debootstrap really does include lsb-release in the initial list of packages to install (this must have changed in a fairly recent Ubuntu release).

I guess this doesn't stop someone from explicitly blacklisting or removing the lsb-release package. I just tried to remove it myself and while this also removes ubuntu-minimal no other adverse side effects are apparent, in other words the removal of lsb-release is still allowed.

I guess this explains how the Docker image reported earlier is missing the package and program. If I were to venture a guess the authors of the Docker image didn't want the lsb-release package in there because it depends on a Python interpreter which would blow up the size of the Docker image...

Of course @FooBarQuaxx is in fact running Python in that same Docker image (apparent from the issue description). Just out of curiosity some questions for @FooBarQuaxx:

You can resolve this issue on your end by installing the lsb-release package in your container and your issue description implies you understand this, so why did you create this issue instead of just installing lsb-release? I ask because I assume you also installed apt-mirror-updater in your container, so you're already running some installation commands... If would be good to know if there are specific reasons to avoid the lsb-release package and lsb_release program apart from "they're not included in minimal Docker images".
Did you install the Python interpreter in the container or did it come pre-installed? I ask because I'm curious to find out if my reasoning for why the lsb-release package isn't installed is correct. In Ubuntu 18.04 the lsb-release package is a dependency of the ubuntu-minimal package and both were installed by debootstrap when I tried, and Docker images are bootstrapped using debootstrap, so it seems to me that the authors of the Docker image specifically made sure that the lsb-release package wasn't part of their Docker image. The only reason I can think of is to minimize image size.

Answer 4 · 2018-10-07T17:16:06.000Z

Short recap: The issue description clearly states that a "vanilla Ubuntu 18.04 docker image" doesn't include the lsb-release package but I found that a minimal Ubuntu 18.04 installation created using debootstrap does in fact include lsb-release.

Out of curiosity I uninstalled the package in my Ubuntu 18.04 chroot and ran grep -r bionic /etc to see if the distribution codename is available anywhere. This search matched the file /etc/lsb-release:

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"

$ dpkg -S /etc/lsb-release 
base-files: /etc/lsb-release

So it seems to me that the file /etc/lsb-release can be relied upon to exist and contain useful information, even if the lsb-release package isn't installed! 🎉 I've since written some code to parse the file and modified the distributor_id and distribution_codename properties to use this new information in preference to running the lsb_release command. I've tested this new code in Ubuntu 14.04, 16.04 and 18.04 chroots (after explicitly uninstalling the lsb-release package 😛) and it works fine.

Answer 5 · 2018-10-07T19:12:54.000Z

I've just released executor 21.1 which first tries to parse /etc/lsb-release and falls back to executing /usr/bin/lsb_release only when /etc/lsb-release isn't available or can't be parsed.

I'm confident this fixes the apt-mirror-updater issue you reported, but because I don't know which exact Docker image you used I can't be 100% sure (I did reproduce the problem in Ubuntu 14.04, 16.04 and 18.04 chroots).

Please try out the new version and by all means let me know if this doesn't resolve the problem for you! (feel free to reopen this issue in that case)