macOS-11 CI/CD build error: remnant /usr/local/bin/2to3
bscottm opened this issue · 9 comments
The cause for the recent CMake-based CI/CD build failures is a remnant /usr/local/bin/2to3 from Python, which is detected by brew when dependencies are installed and updated. Seems to only impact the macOS-11 build. (The makefile only builds for macOS-12, so it never encounters this problem.)
There are a few ways to fix this:
- Drop macOS-11 as a target build platform for prebuilt binaries.
brewemits warnings about macOS-11 obsolescence, so maybe it's time to drop macOS-11. grepforbrew link --overwritesuggestions frombrew, then execute to overwrite the errant links (potential security issue here.)- Individually add commands to the
.travis/deps.shscript and maintain as newbrew linkissues arise.
Which is the preferred option going forward?
Why does 2to3 enter into the picture? Avoiding that would be a good thing.
@pkoning2: Here's the relevant output from .travis/deps.sh osx that installs dependencies via brew for macOS. The output seems to hint that /usr/local/bin/2to3 originated from an Apple pre-installed framework.
.travis/deps.sh has been the "go-to" script for dependencies for a considerable time... and there doesn't appear to be a way to avoid /usr/local/bin/2to3 on macOS-11 (removing it a priori is a pretty hostile thing to do if you run it on your laptop/host.)
==> ./configure --enable-ipv6 --datarootdir=/usr/local/Cellar/python@3.11/3.11.6
==> make
==> make install PYTHONAPPSDIR=/usr/local/Cellar/python@3.11/3.11.6
==> make frameworkinstallextras PYTHONAPPSDIR=/usr/local/Cellar/python@3.11/3.11
==> /usr/local/Cellar/python@3.11/3.11.6/bin/python3.11 -m venv /private/tmp/pyt
==> /private/tmp/pythonA3.11-20231010-4390-1lefa9a/Python-3.11.6/whl_build/bin/p
==> /private/tmp/pythonA3.11-20231010-4390-1lefa9a/Python-3.11.6/whl_build/bin/p
==> /private/tmp/pythonA3.11-20231010-4390-1lefa9a/Python-3.11.6/whl_build/bin/p
==> /private/tmp/pythonA3.11-20231010-4390-1lefa9a/Python-3.11.6/whl_build/bin/p
==> /private/tmp/pythonA3.11-20231010-4390-1lefa9a/Python-3.11.6/whl_build/bin/p
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/2to3
Target /usr/local/bin/2to3
already exists. You may want to remove it:
rm '/usr/local/bin/2to3'
To force the link and overwrite all conflicting files:
brew link --overwrite python@3.11
To list all files that would be deleted:
brew link --overwrite --dry-run python@3.11
Possible conflicting files are:
/usr/local/bin/2to3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3
/usr/local/bin/2to3-3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3-3.11
/usr/local/bin/idle3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/idle3
/usr/local/bin/idle3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/idle3.11
/usr/local/bin/pydoc3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/pydoc3
/usr/local/bin/pydoc3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/pydoc3.11
/usr/local/bin/python3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3
/usr/local/bin/python3-config -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3-config
/usr/local/bin/python3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11
/usr/local/bin/python3.11-config -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11-config
Well, of course for any distributed package to assume /usr/local exists at all, much less expect it to contain Python bits like 2to3, is utterly broken. What is .travis and where does this assumption come from? Not from Apple, I would expect; they use /Library and /System/Library and paths like that.
Perhaps this comes from Homebrew. If so, that would be a major bug. Our answer should be to work around that bug. Among other things, that should mean not using 2to3; there is no reason to use that.
@pkoning2: .travis/deps.sh comes from us. It's the script that installs SDL2, FreeType, PCRE, zlib, et. al., in the CI/CD environment, and is generally useful for newbies to ensure that all dependency libraries are installed before building open-simh. It takes a single argument: linux (Debian and Ubuntu, actually), osx (self explanatory) and mingw64/ucrt64 (MinGW64).
This appears to be a recent change to Github's macOS-11 CI/CD image -- we didn't have this problem with the image until recently (last week or so?)
- It is not a Homebrew problem. It's a CI/CD image "feature".
- macOS-11 is support is deprecated. See the output below, which basically says, "You is on your own."
What this means for open-simh: We need to work around the CI/CD image's "feature" that symlinks 2to3 and other Python binaries into /usr/local/bin.
It'll take some experimentation on my part to figure out how to conditionally work around or remove the offending symlinks from within the CI/CD YAML workflow script, if that's the direction open-simh decides to go. The other option is to remove macOS-11 from the build matrix.
Warning: You are using macOS 11.
We (and Apple) do not provide support for this old version.
It is expected behaviour that some formulae will fail to build in this old version.
It is expected behaviour that Homebrew will be buggy and slow.
Do not create any issues about this on Homebrew's GitHub repositories.
Do not create any issues even if you think this message is unrelated.
Any opened issues will be immediately closed without response.
Do not ask for help from Homebrew or its maintainers on social media.
You may ask for help in Homebrew's discussions but are unlikely to receive a response.
Try to figure out the problem yourself and submit a fix as a pull request.
We will review it but may or may not accept it.
I'm confused. Who wrote the command line that links to /usr/local/bin? Whoever did that is wrong and should fix it.
@pkoning2: The symlink is pre-installed in the macOS-11 CI/CD image. It exists before we install dependencies via brew. It's only when we invoke brew that the issue comes to light (because brew really insists on updating Python.)
I suspect that asking Github to fix something they're going to obsolete is a heavy lift. Just my speculation.
Ok, so can we inject something to remove the wrong symlinks before brew is invoked? If not, it may be the only real option is to remove the MacOS 11 case from the CI/CD machinery. That's not equivalent to deprecating it entirely, it would just mean not testing it because of Github defects.
I'll experiment with adding code to the .github/workflows/cmake-build.yml recipe to remove these symlinks before brew. Not sure how much of the open-simh user base relies on macOS-11 or its prebuilt binaries; if removed, though, we'll find out soon enough... (My preference: Keep macOS-11 around until declared obsolete, since I tend to favor breadth across platforms to identify problems.)
FWIW: Looks like the macOS-11 image was updated on or about 20230921.4 according to the GH tagging.