tox.ini: Add local-macos-nohomebrew environments, deployment targets 10.14, 10.15, 11.1
mkoeppe opened this issue · 49 comments
This will do a "best effort" isolation to avoid using a homebrew installation in /usr/local for anything except bootstrapping and unpacking xz tarballs + a little trick for liblzma.h, which is missing on macOS.
To test:
$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-nohomebrew-python3_xcode
It can then be checked that no libraries from /usr/local leak in, using
find prefix/lib -name "*.so" -o -name "*.dylib*" -exec otool -L {} \;
(Building R is disabled because the R package build still pokes around in /usr/local and finds libintl and liblzma.)
To set specific deployment targets, use:
$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-10.14-nohomebrew-python3_xcode -- config.status
$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-10.15-nohomebrew-python3_xcode -- config.status
$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-11.1-nohomebrew-python3_xcode -- config.status
... and with python3.7 from XCode:
$ tox -e local-macos-10.14-nohomebrew-python3_xcode-python3.7 -- config.status
(Remove config.status from the end of these lines to have the actual build done.)
These environments allow building Sage for older macOS versions, for example for building a 10.14 distribution on a 10.15 machine. The wheels are then correctly tagged, for example pplpy-0.8.6-cp38-cp38-macosx_10_14_x86_64.whl.
Cannot go in the opposite direction (building for 11 on a 10.15 machine) -- because pip will refuse to install a previously built wheel that is tagged for 11.)
Depends on #31552
Depends on #31562
Depends on #31409
Depends on #31584
CC: @jhpalmieri @zlscherr
Component: porting
Author: Matthias Koeppe
Branch/Commit: ed1a089
Reviewer: John Palmieri
Issue created by migration from https://trac.sagemath.org/ticket/31567
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
87e2dba | tox.ini: Add local-macos-nohomebrew and configuration facors macos-{10.14,10.15,11.1} |
Commit: 87e2dba
Author: Matthias Koeppe
Description changed:
---
+++
@@ -1,3 +1,16 @@
This will do a "best effort" isolation to avoid using a homebrew installation in `/usr/local` for anythings except bootstrapping and perhaps unpacking xz tarballs.
+To test:
+```
+$ tox -e local-macos-nohomebrew-python3_xcode -- config.status
+```
+
+To set specific deployment targets, use:
+
+```
+$ tox -e local-macos-10.14-nohomebrew-python3_xcode -- config.status
+$ tox -e local-macos-10.15-nohomebrew-python3_xcode -- config.status
+$ tox -e local-macos-11.1-nohomebrew-python3_xcode -- config.status
+```
+Description changed:
---
+++
@@ -14,3 +14,7 @@
$ tox -e local-macos-11.1-nohomebrew-python3_xcode -- config.status
```
+These environments allow building Sage for older macOS versions, for example for building a 10.14 distribution on a 10.15 machine. The wheels are then correctly tagged, for example `pplpy-0.8.6-cp38-cp38-macosx_10_14_x86_64.whl`.
+
+Cannot go in the opposite direction (building for 11 on a 10.15 machine) -- because pip will refuse to install a previously built wheel that is tagged for 11.)
+Branch pushed to git repo; I updated commit sha1. New commits:
1ed9302 | tox.ini: Add macos-{python3_xcode,nohomebrew}-python{3.7,3.8} |
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
0458f23 | tox.ini: Add macos-{python3_xcode,nohomebrew}-python{3.7,3.8} |
Description changed:
---
+++
@@ -1,4 +1,4 @@
-This will do a "best effort" isolation to avoid using a homebrew installation in `/usr/local` for anythings except bootstrapping and perhaps unpacking xz tarballs.
+This will do a "best effort" isolation to avoid using a homebrew installation in `/usr/local` for anything except bootstrapping and unpacking xz tarballs + a little trick for liblzma.h, which is missing on macOS.
To test:
This is best tested together with #31552, #31562, which do some of the necessary isolation work.
More isolation work is needed at least for pillow and freetype - as observed in #31396 comment:32
Description changed:
---
+++
@@ -14,6 +14,13 @@
$ tox -e local-macos-11.1-nohomebrew-python3_xcode -- config.status
```
+... and with python3.7 from XCode:
+
+```
+tox -e local-macos-10.14-nohomebrew-python3_xcode-python3.7 -- config.status
+```
+(Remove `config.status` from the end of these lines to have the actual build done.)
+
These environments allow building Sage for older macOS versions, for example for building a 10.14 distribution on a 10.15 machine. The wheels are then correctly tagged, for example `pplpy-0.8.6-cp38-cp38-macosx_10_14_x86_64.whl`.
Cannot go in the opposite direction (building for 11 on a 10.15 machine) -- because pip will refuse to install a previously built wheel that is tagged for 11.)Description changed:
---
+++
@@ -17,7 +17,7 @@
... and with python3.7 from XCode:
```
-tox -e local-macos-10.14-nohomebrew-python3_xcode-python3.7 -- config.status
+$ tox -e local-macos-10.14-nohomebrew-python3_xcode-python3.7 -- config.status
```
(Remove `config.status` from the end of these lines to have the actual build done.)
Branch pushed to git repo; I updated commit sha1. New commits:
236ea3c | tox.ini (macos-nohomebrew): Pass compiler configuration to configure using variables other than CC, CXX. |
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
92db4e4 | tox.ini (macos-nohomebrew): Pass compiler configuration to configure using variables other than CC, CXX. |
Description changed:
---
+++
@@ -3,15 +3,21 @@
To test:
```
-$ tox -e local-macos-nohomebrew-python3_xcode -- config.status
+$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-nohomebrew-python3_xcode
```
+It can then be checked that no libraries from /usr/local leak in, using
+
+```
+find prefix/lib -name "*.so" -o -name "*.dylib*" -exec otool -L {} \;
+```
+(Building R is disabled because the R package build still pokes around in /usr/local and finds libintl and liblzma.)
To set specific deployment targets, use:
```
-$ tox -e local-macos-10.14-nohomebrew-python3_xcode -- config.status
-$ tox -e local-macos-10.15-nohomebrew-python3_xcode -- config.status
-$ tox -e local-macos-11.1-nohomebrew-python3_xcode -- config.status
+$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-10.14-nohomebrew-python3_xcode -- config.status
+$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-10.15-nohomebrew-python3_xcode -- config.status
+$ EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-11.1-nohomebrew-python3_xcode -- config.status
```
... and with python3.7 from XCode:Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
737b21c | src/sage/stats/r.py: Mark all 2 doctests in this file as # optional - rpy2 |
4403924 | src/sage/interfaces/r.py: Mark all tests # optional - rpy2 |
405ebb9 | More # optional - rpy2 |
b7c1576 | Merge tag '9.3.beta8' into t/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional |
ce3b35f | More # optional - r |
49c10c1 | Revert "build/pkgs/{r,rpy2}: Downgrade to optional" |
a67300b | Merge #30383 |
f0a5fb5 | configure.ac: Add option --disable-r |
f04c134 | Merge branch 't/30383/new_package_type__optional_enabled_by_default' into t/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional |
7966b66 | Merge #31409 |
Branch pushed to git repo; I updated commit sha1. New commits:
b5edde2 | tox.ini (macos-nohomebrew): Fix typo in PILLOW_... |
Now pillow does not find zlib
Branch pushed to git repo; I updated commit sha1. New commits:
dffcdbc | tox.ini (macos-nohomebrew): Fix up pillow build by setting ZLIB_ROOT |
pillow still finds various libraries in /usr/local because of very creative discovery code that finds everything in /usr/local because we told it that it can have /usr/local/opt/xz/include - see _add_directory (https://github.com/python-pillow/Pillow/blob/8.1.x/setup.py#L197)
Branch pushed to git repo; I updated commit sha1. New commits:
3779a25 | tox.ini (macos-nohomebrew): Disable more libraries to remove dependencies on homebrew |
Fixed pillow. Now I can build a wheel in #31396 that does not have any dependencies on shared libraries in /usr/local.
For what it's worth, I see a few suspicious items in the log files. (This is on a system with homebrew and with source .homebrew-build-env in my shell init file. That may be causing some of these.)
- cvxopt includes flags
-L/usr/local/liband-I/usr/local/includewhen callinggcc - pillow says
Appending path /usr/local/Cellar/xz/5.2.5/includeand then thegccflag-I/usr/local/Cellar/xz/5.2.5/include.
The xz include is on purpose -- see comment added in tox.ini
Replying to @jhpalmieri:
For what it's worth, I see a few suspicious items in the log files. (This is on a system with homebrew and with
source .homebrew-build-envin my shell init file. That may be causing some of these.)
Yes, this a good setting for testing this ticket. The build using tox is designed to be isolated from the effects of .homebrew-build-env.
- cvxopt includes flags
-L/usr/local/liband-I/usr/local/includewhen callinggcc
This is probably https://github.com/cvxopt/cvxopt/blob/master/setup.py#L55 in combination with a bug in our build system - #31584.
Reviewer: John Palmieri
Okay, this looks good to me.
Thanks!
By the way, should commands like make distclean also clean the .tox directory?
I think it should not because .tox is not a "build artifact" - if this reasoning makes sense.
That makes sense. Maybe I need to learn more about tox. What command should I use if I want to restart EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-nohomebrew-python3_xcode from scratch? Just delete .tox/local-macos-nohomebrew-python3?
Replying to @jhpalmieri:
That makes sense. Maybe I need to learn more about tox. What command should I use if I want to restart
EXTRA_CONFIGURE_ARGS="--disable-r" tox -e local-macos-nohomebrew-python3_xcodefrom scratch? Just delete.tox/local-macos-nohomebrew-python3?
That works, but you can also pass the -r option to tox, which will start from scratch.
Great, thanks!
Changed branch from u/mkoeppe/tox_ini__add_local_macos_nohomebrew_environments to ed1a089