[Question] Any plans for musl based distros?
squeaky-pl opened this issue · 21 comments
What are the plans if any for musl based distributions? Musl based distributions are popular in container world because they are much lighter. Docker announced that they will be redoing all their official images to be Alpine Linux based. Will there be a separate tag?
A better place for this discussion would be the distutils-sig mailing list, since that's where any such plan would need to be discussed and approved, and also would reach a much larger audience of people who might be interested in doing the work. I'll answer below in any case, but that's probably the best place for follow-ups, so I'll also close this bug :-).
Anyway: there aren't any concrete plans for a musllinux tag, no. We're aware of Docker and Alpine Linux, and there's been some discussion of also creating distro-specific tags, which would make it possible to define wheels targeting Alpine Linux specifically. (Are there any other musl-based distributions that people care about in practice? Is there sufficient commonality between different musl-based distributions that a single wheel can reasonably target several at once?)
My personal guess is that Docker's announcement about changes to their official images won't affect Python much -- the official Python images have always come in both musl and glibc options, so if you want manylinux wheels you can and should pick the latter; I doubt they'll suddenly stop providing this option.
Obviously having some sort of musl wheel option would also be nice, but this requires someone to define the scope, write up a proposal, etc. If you're interested in helping with this then @natefoo might be someone to talk to.
@njsmith is it worth reopening this now that there are official alpine python images? libm in particular is really hard to get on alpine.
@abe-winter: I think everything in my previous reply still applies. I think it would be great if someone wants to work in this, but the text above already says what you need to do to move things forward, and it doesn't involve this repository.
@njsmith fair enough -- thanks
Hi folks,
sorry to "reopen" this issue, but I would like to know how to help on alpine/musl support for wheel.
We have a lot of python containers based on Alpine and I would like to use wheel.
I'm ok trying to help but I don't known where to start. If someone can guide me I'll be happy to help on that.
Thanks
You'd need to write a PEP defining a new wheel "platform tag" for alpine. The main problems are (1) you need to make sure that pip can look at the system it's running on, and a wheel, and determine whether the wheel and the system are compatible, (2) you need some way to figure out which wheels should receive this tag, (3) you need some way to reliably build these wheels. There are some judgement calls to make along the way -- do you want to target all musl systems, or specifically alpine systems, or ...? How do you recognize when you have a wheel for alpine, but it requires a newer version of alpine than pip is running on? This may require working with the alpine or musl developers to figure out what ABI guarantees they actually make, which libraries you can assume are available, etc.
PEPs are discussed on the distutils-sig mailing list; you should probably post drafts regularly to get feedback, but expect that you'll have to do the driving to make this happen.
PEP 1 and PEP 12 gives details on how you write a PEP. You'll probably want to refer to PEP 491 for details on wheels, and maybe PEP 513 is a useful example to refer to too.
@njsmith (1) Is there a way of achieving this without writing new PEP? It seems to be an overkill. Would it be possible to extend platform detecting function instead? Because there few manylinuxes available.
As for (2) and (3), that's quite easily to do – Travis CI with alpine build, creating wheels for all popular packages will work in my opinion. I'm gonna start working on this soon.
@lyssdod Well, we need to figure out what tag is used for alpine wheels, we need to figure out how pip can tell when it's running on alpine, and we need to figure out a way for auditwheel or similar to create these wheels. Then we need to write down what we've figured out, so that there's a reference and everyone's on the same page. That's what a PEP is. We can't just skip writing it down, no.
@njsmith OK! I will look into examples
Also see pypa/packaging-problems#69
This might be relevant: https://natemcmaster.com/blog/2016/05/19/nuget3-rid-graph/
NuGet's notion of RIDs/runtimes serves the same purpose and is less strict (longest prefix match:ish).
Kind of resurrecting the topic; I've spent some time and made an MVP, but discussion eventually died without replies from the maintainers. Does anybody know where python packaging discussion is happening for real ? Apparently not in the mailing lists
@lyssdod - sorry - maybe I missed it - did you point to the MVP / discussion here?
Does anybody know where python packaging discussion is happening for real ? Apparently not in the mailing lists
It's mostly moved to discuss.python.org, in the "packaging" category.
It's also appreciated if you can send a ping if you're not getting replies – folks are busy and stuff gets missed sometimes.
@matthew-brett The distutils-sig thread was back in February at https://mail.python.org/archives/list/distutils-sig@python.org/thread/H3323AXRRLJAYOY2XZKS74IOUQMJUOYD/
That linked to Alex's musl detection PoC at https://gist.github.com/lyssdod/f51579ae8d93c8657a5564aefc2ffbca
However, Alex's last set of questions in that thread never actually got an answer :(
Thanks - then I think Nathaniel's answer above is the right one - and he is one of the people he's encouraging you to ping, to remind us to answer :)
@ncoghlan @matthew-brett thanks! that's exactly what I was meaning. I will repost it there then
FYI there's now a musllinux tag https://www.python.org/dev/peps/pep-0656/ but it's not yet supported by the ecosystem (I only saw one PR adding it to packaging).
Looks like support is becoming more commonplace:
- pypa/cibuildwheel#768
- https://github.com/pypa/cibuildwheel/releases/tag/v2.2.0a1
- pypi/warehouse#9982
- pypa/packaging#411
Perhaps it may be time to revisit?