all: require Linux 3.2 kernel for Go 1.24
rsc opened this issue · 34 comments
In 2021 in #45964 we moved the minimum Linux kernel version to 2.6.32.
There is a good summary from @baryluk about what we use from newer kernels at #60792 (comment).
@mdempsky points out in #60792 (comment) that Rust and glibc decided on a minimum kernel version of 3.2 last year.
Adopting 3.2 as Go's minimum seems reasonable given the rest of the Linux ecosystem doing the same.
I propose we announce this in Go 1.23 and make the change in Go 1.24. That will still be a year or so behind glibc and Rust, and the announcement will create time for people to speak up if they are affected and cannot upgrade.
Really everyone should be on newer kernels for security updates anyway.
I'm ambivalent about this trend in general. For Go, it also pushes a bit against the compatibility promise.
I've been burned multiple times by manufacturers dropping support for old hardware, requiring expensive upgrade purchases that provide no return benefit to me. I still run a 32-bit x86 Mac in the lab because without it I cannot control an important piece of kit that can only be interfaced to an old mac OS version. I can't even connect it to the network for fear some update will render my equipment non-working.
While I appreciate the implied - if uncertain - value of "newer is safer, newer is better", what is the actual cost to the project of letting Go binaries run on old OSes? That seems a better way to evaluate this proposal.
The cost to the project in general is continued maintenance and testing of code paths working around the lack of features from the (relatively) newer kernels. We don't do a great job of testing Go on these old kernels as it is. It probably only kind of works.
I should have mentioned that Linux 3.2 was released 12 years ago. It is not bleeding edge by any means. According to https://en.wikipedia.org/wiki/Linux_kernel_version_history, the final 2.6.x kernel was 2.6.32.71 in 2016.
We are talking about dropping support for kernels that are very old.
Talking to @FiloSottile about #66821, it sounds like moving to 3.17 (~2014) as the minimum would let us avoid needing to open /dev/urandom ever, because 3.17 added getrandom(2). That would be a better choice than 3.2.
This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group
In the way of additional data points, a few popular and older distribution versions and their associated kernel versions (all of which IMO point in favor of this proposal at 3.2 and even at 3.17 👍):
- Debian 10 (Buster): 4.19
- Ubuntu 20.04 (focal): 5.4 (you have to go all the way back to 14.04 to get 3.13 / incompatible)
- CentOS 7: 3.10 (but, 7 is fully EOL on 2024-06-30 in two short months)
- CentOS 8: 4.18 (EOL 2021-12-31)
- Fedora 38: 6.2
- OpenWrt 22.03: 5.10 (EOL 2024-04-11)
It's a bit hard to browse, but even more aggregated data at https://repology.org/project/linux/versions
(I dug up this information to convince myself this was sane and figured it was worth sharing 👍 ❤️)
Edit: and see #60792 (comment) for a much longer form of similar data with many more details! 😄 ❤️
Red Hat Enterprise Linux 7 is on kernel 3.10 and it's maintenance ends 30 June 2024. Red Hat offers Extended Lifecycle Support until 30 June 2028. Gcloud CLI supports RHEL7 and it includes Golang binaries. You may want to clarify how long Gcloud CLI plans to support RHEL7.
The RHEL 7.9 OS has a kernel version 3.10.0 and is supported until 2028-06. Changing the minimum version to 3.17 may impact services running on this OS.
https://distrowatch.com/table.php?distribution=redhat&pkglist=true&version=rhel-7.9#pkglist
The RHEL 7 kernel had getrandom backported in 2017, per https://access.redhat.com/errata/RHSA-2017:1842:
BZ - 1330000 - kernel: Backport getrandom system call
So in practice Go programs would continue to work on RHEL 7 if the only 3.10+ feature used is getrandom.
Thanks for the added details. I think we can say 3.10+getrandom is fine too.
Have all remaining concerns about this proposal been addressed?
The proposal is to require Linux 3.17 starting in Go 1.24,
with the exception that Linux 3.10 patched with an added getrandom(2) system call is also permitted.
I think 3.17 (or 3.10+getrandom) is doable and reasonable.
I had a minor concern with Debian 8 codename Jessie, released April 2015, which is still in extended long term support by freexian (paid support, until mid 2025, possibly mid 2026 at most if customers pay for it), because it had 3.16.7.
But even during normal LTS there were backports available with kernel 4.9, and now freexian recommends their backports (due to security updates mostly) maintained by them of kernel 4.19 and 5.10. And anybody still running so old system (initial release 2015-04-26, official EOL 2020-07; and now we are at Debian 12, so many many major releases behind), can just stick to old code, or continue using older Go to maintain any software they run.
So even Debian 8 with ELTS would be fine with this 3.17+ requirement in practice. (Debian 9 codename Stretch, was released with 4.9.30, on 2017-06-18, already out of normal LTS, and ELTS from freexian until 2027).
So, good to go with 3.17
Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group
The proposal is to require Linux 3.17 starting in Go 1.24,
with the exception that Linux 3.10 patched with an added getrandom(2) system call is also permitted.
No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group
The proposal is to require Linux 3.17 starting in Go 1.24,
with the exception that Linux 3.10 patched with an added getrandom(2) system call is also permitted.
Change https://go.dev/cl/588216 mentions this issue: internal/poll,syscall: drop the fallback for accept4 on linux/arm
Change https://go.dev/cl/589015 mentions this issue: doc/next: preannounce a new Linux kernel version requirement
I still support this, but in the interest of being aware of expected breakage, Synology apparently ships Linux 3.10 without getrandom() in recent (current?) firmwares. The "good" news is that apparently even glibc gave up on that, so a C application invoking g_random_int will fatal out, like we plan to do in crypto/rand (#66821). See photoprism/photoprism#4339 (comment) where a Go application was crashing out due to a cgo glibc invocation. They worked around it, but they won't be supported by Go 1.24.
Change https://go.dev/cl/602495 mentions this issue: crypto/rand: remove /dev/urandom fallback and simplify package structure
@bradfitz See the comment #67001 (comment). I seem to recall that you've mentioned Synology before.
We have 10,329 7-day active users using Synology devices with kernels older than 3.17:
- Linux 3.10.108 (7027 active users)
- Linux 3.2.101 (1952 active users)
- Linux 2.6.32.12 (484 active users)
- Linux 3.2.40 (435 active users)
- Linux 3.10.102 (431 active users)
And then we also have 6,026 other active Linux users NOT using Synology running kernels older than 3.17.
So 16,355 active users who would be impacted by this change.
If we need to drop back to 3.2 to match Rust and glibc, with the understanding that it will mean keeping some crypto code that opens /dev/urandom for a while longer (and crashes if that fails), then I'm okay with that if others are. It is a very tiny amount of code, and getrandom appears to be the only thing we were hoping for after 3.2.
I don't see too much point to dropping behind what Rust and glibc now require, though.
And then we also have 6,026 other active Linux users NOT using Synology running kernels older than 3.17.
How many of those are RHEL 7 and derivatives (which backported getrandom)?
Looks like ~4770 active users are RHEL/Centos with some variant of Linux 3.10.0 ...
I wonder how many of these older versions are actually using recent versions of Go? Seems to me that if people don’t or can’t upgrade to newer kernels, maybe they aren’t keeping up to date with Go releases as well… I’m guessing the telemetry data won’t help here?
The challenge is that it isn't just a constraint of the system running Go, but on the systems running the Go programs, so the telemetry won't capture that sample set.
(I still think the change is fine, to be clear, just trying to help explain why it's Complicated and doesn't necessarily relate to systems using the Go toolchain directly very much.)
@meling For a case like Synology, this is users who want to run a new Go program on their Synology system, but it may not be straightforward for them to update that system to a newer kernel.
Change https://go.dev/cl/608175 mentions this issue: crypto/rand: reintroduce urandom fallback for legacy Linux kernels
After discussion with @FiloSottile we've agreed to drop down to Linux 3.2 as the minimum version for future versions of Go, matching glibc and Rust. crypto/rand will have a fallback to open /dev/urandom for randomness (OCLOEXEC, of course) when getrandom returns ENOSYS.
Change https://go.dev/cl/609178 mentions this issue: syscall: always use prlimit for getrlimit/setrlimit on Linux
Now that the decision is to only require Linux 3.2, should we update https://go.dev/doc/go1.23? It still says:
Go 1.24 will require Linux kernel version 3.17 or later, with an exception that systems running 3.10 or later will continue to be supported if the kernel has been patched to support the getrandom system call.
@mvdan Thanks for pointing that out. https://go.dev/cl/611943
Change https://go.dev/cl/611943 mentions this issue: _content/doc/go1.23: we are only going to require Linux kernel 3.2
From our weekly review, this issue is still open and marked as a release blocker just because we need to make sure it ends up in the Go 1.24 release notes.
Change https://go.dev/cl/622015 mentions this issue: doc/next: document the minimum Linux kernel version for 1.24
The release notes have been submitted. Please feel free to re-open this issue if any changes were missed.