Ubuntu-latest workflows will use Ubuntu-22.04
ddobranic opened this issue · 57 comments
Breaking changes
Ubuntu 22.04 is ready to be the default version for the "ubuntu-latest" label in GitHub Actions and Azure DevOps.
Target date
This change will be rolled out over a period of several weeks beginning in October, 3. We plan to complete the migration by December, 15th, 2022.
The motivation for the changes
GitHub Actions and Azure DevOps have supported Ubuntu 22.04 in preview mode since May 2022, and starting from August 2022 Ubuntu 22.04 is generally available for all customers. We have monitored customer feedback to improve the Ubuntu 22.04 image stability and now we are ready to set it as the latest.
Possible impact
The Ubuntu 20.04 image has a different set of software than Ubuntu 22.04. The most significant changes are listed in the table below:
| Tool name | Ubuntu 20.04 | Ubuntu 22.04 | Notes |
|---|---|---|---|
| Ruby toolcache | 2.5.9 2.6.10 2.7.6 3.0.4 3.1.2 |
3.* (latest) | Only latest available version will be cached on start. |
| Ruby | 2.7.0 | 3.* (latest) | The latest available version for each operating system provided in the APT repository is installed. |
| Clang Clang-format Clang-tidy |
10.0.0 11.0.0 12.0.0 |
12.* 13.* 14.* |
Three latest available versions for each operating system provided in the APT repository are installed. |
| GCC GFortran |
9.4.0 10.3.0 - |
9.* 10.* 11.* |
Three latest available versions for each operating system provided in the APT repository are installed. |
| Python | 3.8.10 | 3.10.6 | Fresh updates for Python and tools available for Ubuntu 22 in APT repository. |
| PHP | 7.4.32 8.0.24 8.1.11 |
8.* (latest) | Only one latest officially supported PHP version will be supported. See more here. |
| .NET Core SDK | 3.1.120 3.1.202 3.1.302 3.1.423 5.0.104 5.0.214 5.0.303 5.0.408 6.0.401 |
6.0.401 | Only the latest available version will be installed on start. |
| Node.js | 12.22.12 14.20.1 16.18.0 18.12.0 |
- 14.20.1 16.18.0 18.12.0 |
EOL version had been deprecated. If your use-case requires using any of these versions, consider using tasks to install Node.js on-flight: - actions/setup-node (GitHub Actions) - Node.js Tool Installer (Azure DevOps) |
| PyPy | 2.7.18 [PyPy 7.3.9] 3.6.12 [PyPy 7.3.3] 3.7.13 [PyPy 7.3.9] 3.8.13 [PyPy 7.3.9] 3.9.12 [PyPy 7.3.9] |
3.7.13 [PyPy 7.3.9] 3.8.13 [PyPy 7.3.9] 3.9.12 [PyPy 7.3.9] |
Deprecated all the versions besides the most recent ones. |
| Python toolcache | 2.7.18 3.6.15 3.7.14 3.8.14 3.9.14 3.10.7 |
3.7.14 3.8.14 3.9.14 3.10.7 |
On GitHub Actions, actions/setup-python can install any version on-flight so this change doesn't impact users. |
Platforms affected
- Azure DevOps
- GitHub Actions
Runner images affected
- Ubuntu 18.04
- Ubuntu 20.04
- Ubuntu 22.04
- macOS 10.15
- macOS 11
- macOS 12
- Windows Server 2019
- Windows Server 2022
Mitigation ways
If you see any issues with your workflows during this transition period:
- File an issue in this repository
- Switch back to Ubuntu 20 by specifying the “Ubuntu-20.04" label. We will continue to support Ubuntu 20.
For more information you can take a look at the previous announcement here.
The "Runner images affected" tasks list appears to be incorrect as it targets "macOS" instead and not "ubuntu", is that a mischeck?
Does the ubuntu-22.04 install Firefox as a snap package, or directly, or not at all? I'm seeing test failures with that image, and worry that I'll need to either add a Firefox install to my workflow or maybe pin to ubuntu-20.04 for a while.
Edit: It appears not to be installed at all: https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2204-Readme.md#browsers-and-drivers
Is that an oversight that will be corrected, or is it deliberate and we all need to adjust? Thanks!
20.04 LTS has actually shipped firefox as a snapd feature in it's desktop variant whereas the server variant does not have it at all - but can be looked into.
We're using ubuntu-latest images for Firefox testing today. I tried ubuntu-22.04 in a PR to see what would happen, and our workflow fails to detect Firefox at all. So it looks like we'll have to either pin to ubuntu-20.04 or install Firefox explicitly to avoid issues.
In one of our repos, Ubuntu 22.04 based tests are failing. I am yet to figure out the issue. But the same tests working with 20.04.
The following packages have unmet dependencies:
libunwind-14-dev : Breaks: libunwind-dev but 1.3.2-2build2 is to be installed
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
Error: Process completed with exit code 100.
Hello, @aravindavk!
Here is an explanation why this is happening with your build. I added a comment explaining how to fix the situation.
Thanks @erik-bershel
@ddobranic it seems that Ubuntu Mozilla Team still provides official non-snap ESR packages of Firefox via PPA:
https://launchpad.net/~mozillateam/+archive/ubuntu/ppa?field.series_filter=jammy
Maybe you could coordinate with them, not having any Firefox at all in CI images is quite... limiting.
Is there any updates on Firefox not being included in ubuntu-22.04?
@Josh-Osmanski yes, we are planning to add Firefox. thank you!
@ddobranic The original post mentioned a migration happening automatically between October, 3 & December, 1. We're in the middle of this period. If it's indeed happening, auto-migrated repositories doing Firefox testing may start seeing breakages since Firefox has not been added to the image yet. What's the plan around that?
Can confirm that loads of repositories started to break on our end.
Not shipping previously available packages is breaking in my honest opinion.
Could you update the Node.js entry in the table ^^?
The Node.js Active LTS version 18.12.0 is now cached in both ubuntu-20.04 and ubuntu-22.04. The 16.x version has also been updated.
I think it should look like this:
| Tool name | Ubuntu 20.04 |
Ubuntu 22.04 |
|---|---|---|
| Node.js | 12.22.12 14.20.1 16.18.0 18.12.0 |
- 14.20.1 16.18.0 18.12.0 |
@MikeMcC399 thank you! updated.
The table in https://github.com/actions/runner-images/blob/main/README.md still shows ubuntu-latest as ubuntu-20.04. This is confusing and should probably be fixed, thanks!
@thiell the migration happens gradually, not all current ubuntu-latest resources are using ubuntu-22.04 yet. Once it is done the docs will be updated.
Hey @ddobranic
It seems that ubuntu-22.04 does not have MongoDB installed, Is there any plan to add it as a default package?
@ddobranic Is there any update on adding Firefox as a browser to Ubuntu latest?
@chuma9615 there is still no official ubuntu 22.04 release as far as I can see, only preview
This updated image broke my CI WASM tests which make use of Firefox. I know this is incoming, but it's been a while since it weas stated that this is planned but things are still breaking. Is there any update on this?
@Bluefinger ub22.04 with firefox will be released this week, it is already in main
Awesome! Good to know and thanks for the update
Are you still planning to complete the migration of Ubuntu-latest to use Ubuntu-22.04 by Dec 1, 2022? The repos which I am involved in are still using 20.04 when they run with runs-on: ubuntu-latest specified in the GitHub workflow.
@MikeMcC399 we have stretched the migration to ease it for all the customers and make it more smooth
we have stretched the migration to ease it for all the customers and make it more smooth
Thank you for your feedback! In that case perhaps you could update the original post with your new target date for completion of the migration? If you don't have a concrete new date, please at least mention that the completion has been extended past Dec 1, 2022 so that readers are informed.
@MikeMcC399 indeed, updated :)
I see that you changed the completion date for the migration to March 1 (2023 assumed). That is quite a significant change of 3 months from the previous Dec 1, 2022. Thank you for the update!
I just had a workflow that uses ubuntu-latest end up using ubuntu-22.04 today. In my time zone, December 1 is tomorrow, but whatver. I'm just confused when this is rolling out given that the issue is still open and the previous comment about postponed migration.
@2bndy5 it is not postponed, but rather stretched, i.e. it is in progress and ubuntu-latest will get updates to ubuntu-22.04
After I posted, I realized the deadline was a end-of-migration date, not a hard overnight change. For now, I'm using ubuntu-22.04 to keep the workflow consistent during migration. Thanks for responding!
The GitHub Changelog from Dec 1, 2022 says:
"Larger runner workflows using the ubuntu-latest runner label will soon run on Ubuntu-22.04.
Ubuntu-22.04 is now the default version for the ubuntu-latest label for GitHub Actions standard runners workflows. Larger runners will now use the Ubuntu-22.04 as the -latest version starting 15 December 2022."
How is it possible to determine if a runner belongs to the category "larger runners"? What is the metric for this evaluation?
@MikeMcC399 large runners are paid ones.
large runners are paid ones.
Thank you so much for the explanation!
Can I suggest that the link to large runners be added to the GitHub Changelog from Dec 1, 2022? I didn't realise that the term "larger runners" in the blog had a special meaning.
The inclusion of only .NET 6 and 7 on 22.04 contradicts the statement in the top-level README
2 latest LTS versions and 1 latest version. For each feature version only latest patch is installed
According to that, 3.1 should also be included, being the second most recent LTS version
Python 3.6 isn't available on Ubuntu 22.04. Any plans to make that available / backport?
@kynan python 3.6.x has been EOL since December '21, so there is no plans to add it to 22.04
FYI Our C++ CI tests started failing on Linux in Azure DevOps from 1 Dec. when ubuntu-latest switched from 20.04 to 22.04. This failure happened whenever the test executables were started:
symbol lookup error: /home/vsts/work/1/s/bin/os/linux_x64/test/working/../../../../../build/os/linux_x64/packages/intelmkl.devel.linux-x64.2019.5.281/lib/native/linux-x64/libmkl_intel_thread.so: undefined symbol: omp_get_num_procs
A colleague fixed this by adding this to the pipeline yml:
sudo apt install --no-upgrade libomp-dev
Will we have to do this from now on? Or will you be adding libomp-dev to the ubuntu-latest (22.04) image?
FYI, the error message showing up in AzDO builds is misspelled (unless the underlying name is changing to ubntu):
##[warning]ubuntu-latest pipelines will use ubntu-22.04 soon. For more details, see https://github.com/actions/runner-images/issues/6399
@mechie it depends on a pipeline type, azdo still supports ubuntu-22.04, say, in vmImage
@mechie it depends on a pipeline type, azdo still supports
ubuntu-22.04, say, invmImage
What? He is telling you there is a typo in the warning and you are saying yes, there is a pipeline type that will accept the typo?
##[warning]ubuntu-latest pipelines will use ubntu-22.04 soon. For more details, see #6399
I really wish Azure DevOps did not warn on this sort of stuff, it should be info: the ubuntu-latest tag is for running the latest version of Ubuntu, why are you warning me that I am getting the latest version of Ubuntu? That's what ubuntu-latest means, that's why I use it.
Weirdly it gets it right sometimes so hopefully that is a new agent image being rolled out...
The Original Post says:
Target date
This change will be rolled out over a period of several weeks beginning in October, 3. We plan to complete the migration by March, 1st, 2023.
This seems more like a several MONTH rollout. Will we endure several months of warnings that ubuntu-latest workflows will use ubuntu-22.04 soon.? (That's not "soon".)
I apologize for being grumpy, but unnecessary warnings make me unhappy.
Update: Why am I unhappy? I have 40 occurrences of ...-latest in my small project. I'm not aware of a facility in GHA that would allow me to specify a variable that contains ubuntu-20.04 (or ubuntu-22.04) and use it globally. So I'm stuck either living with the warnings or globally editing that string a load of times...
Update: Why am I unhappy? I have 40 occurrences of
...-latestin my small project.
We feel the same pain. It would be much less annoying if Azure Pipelines can group all the issues with the same error. And/or by default just collapse all the errors.
.NET Core SDK : "Only the latest available version will be installed on start.". so version 7 only?
@mikeblakeuk it is already more than one: https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2204-Readme.md#net-core-sdk
One of my actions fails on 22.04 because a C header cannot be found (#include <omp.h>). However, it still works with 20.04, no code changed.
What's the best way to investigate?
Google it? omp.h not found ubuntu yields this as the second result.
This is now also the case on Ubuntu 22.04 (which using
clang-14). More generic install is
sudo apt install libomp-dev
Not mentioned in the table above, but Ubuntu 22.04 ships with OpenSSL 3 by default. I thought that pinning to 20.04 should solve it.
-runs-on: ubuntu-latest
+runs-on: ubuntu-20.04But that wasn't a drop-in replacement, I immediately hit a build failure due to glibc: error: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.33' not found
I did not dig into this any further, and instead I will just be upgrading my systems to use OpenSSL 3.
Running docker-compose runs into a permission issue.
[2326] Failed to execute script docker-compose
Traceback (most recent call last):
File "docker-compose", line 3, in <module>
File "compose/cli/main.py", line 81, in main
File "compose/cli/main.py", line 203, in perform_command
File "compose/metrics/decorator.py", line 18, in wrapper
File "compose/cli/main.py", line 970, in run
File "compose/cli/main.py", line 1460, in run_one_off_container
File "compose/cli/main.py", line 1586, in call_docker
File "subprocess.py", line 339, in call
File "subprocess.py", line 800, in __init__
File "subprocess.py", line 1551, in _execute_child
PermissionError: [Errno 13] Permission denied: '/opt/hostedtoolcache/docker-stable/17.9.0-ce/x64/docker'
ubuntu-latest has finally been switched to ubuntu 22.04 for both GitHub Actions and Azure DevOps!
It appears that this version doesn't support .NET 7 builds. Is that accurate or is there a way around the issues reported by the pipelines for .NET 7 projects built on ubuntu-latest right now?
@sjd2021 it's what is installed by default, you can just add a UseDotNet-task (ex for DevOps) to install dotnet7 if you'd like
@kamnyborg I've done so, but that didn't seem to shake off whatever is limiting the MSBuild version to 16.x when it needs to be 17.x.
Is there any update on the version of PowerShell? Just double checking whether 7.2.7 is still the version as indicated here. Thank you
What about gnome?