relocatable wheel version of package sage_conf

Question

relocatable wheel version of package sage_conf

mkoeppe opened this issue 5 years ago · 269 comments

This version of sage_conf is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making SAGE_ROOT (and thus SAGE_LOCAL=$SAGE_ROOT/local) relocatable using Marc Culler's symbolic link surgery (the method proposed in #31076, using SAGE_ROOT=/var/tmp/sage-... and a symlink).

The sage script invokes sage-config to determine SAGE_ROOT and SAGE_LOCAL. In the version of sage-config supplied by this version of sage_conf, we ensure that the symlink from /var/tmp/sage-.... to the actual install location is set.

So far, a wheel has been built for XCode python 3.8 on macOS 10.15. The wheel is 670MB in size, an order of magnitude above the standard file size limit on PyPI; but a file size limit increase (requested in pypi/support#985) has kindly been granted by the PyPI team.

The wheel declares dependencies (install-requires) to all Python packages in the Sage distribution that have extension modules. The dependencies are specific using @ to URLs on
https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2, where I have uploaded the binary wheels. Unfortunately, by PyPI policy, such @ references are not allowed for packages on PyPI.
Hence, the pip invocation needs to use a URL.

(See also https://twitter.com/mkoeppe_math/status/1378860285537054723)

Instructions for testing:

Make sure that the bin directory of the user installation scheme is in PATH. For example, on macOS:

$ export PATH=$HOME/Library/Python/3.8/bin:/usr/bin:/bin:/usr/sbin:/sbin

Install the wheel in the user scheme:

$ /usr/bin/python3 -m pip install -U --user  https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sagemath_standard-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl

Run the installed sage.

$ sage

Procedure for building the sage_conf wheel and compatible wheels:

For building wheels on macOS Catalina that uses XCode python 3.8:

$ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
(tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel --plat-name macosx_10_15_x86_64)

or

tox -e local-macos-10.15-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

This creates (uploaded to https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2)

src/pkgs/sage_conf-relocatable/dist/sage_conf-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl
src/pkgs/sage_conf-relocatable/sage_root/venv-cpython-38-darwin-macosx_10_15_x86_64/var/lib/sage/wheels/*.whl

Build for macOS Big Sur (requires a machine running Big Sur or Monterey) (uploaded to https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2)

tox -e local-macos-11.1-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

Build for macOS Monterey (requires a machine running Monterey)

tox -e local-macos-12.1-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

Build for Linux (does not work yet):

$ TARGETS_PRE=Makefile tox -e docker-manylinux-2_24-minimal-python3.9 -- sage-wheels

Follow-up steps:

also build a sagemath-standard wheel
build wheels for manylinux using tox -e docker-manylinux....
- wheel is much too big: 2.3GB
- auditwheel complains: RuntimeError: Invalid binary wheel, found the following shared library/libraries in purelib folder:; so we should replace our current abuse of package data (install_data) by platlib (install_lib). pypa/packaging-problems#542 (comment): subclass build_ext instead of build_py
disable building the dependencies of the sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible -- for example by storing a tar file sage_root.tar (with symlinks) instead of including the sage root as a directory - https://docs.python.org/3/library/tarfile.html#tarfile.open
#31602: split out sage_root from sage_conf and use install-requires in the sage_conf sdist/wheel with @ links to GH releases to the sage_root wheel and the built non-any wheels.

Depends on #33817

CC: @culler @kliem @dimpase @jhpalmieri @slel @kiwifb

Component: relocation

Author: Matthias Koeppe

Branch/Commit: u/mkoeppe/relocatable_wheel_version_of_package_sage_conf @ 1b2d4be

Reviewer: Julian Rüth

Issue created by migration from https://trac.sagemath.org/ticket/31396

Answer 1 · 2021-02-15T03:32:14.000Z

Branch: u/mkoeppe/relocatable_wheel_version_of_package_sage_conf

Answer 2 · 2021-02-15T06:36:08.000Z

Commit: 9573b7c

Answer 3 · 2021-02-15T06:36:08.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`952fb9a`	`Fix up package_data`
`9573b7c`	`Closer to working`

Answer 4 · 2021-02-15T06:40:47.000Z

Description changed:

--- 
+++ 
@@ -2,3 +2,6 @@
 
 The `sage` script invokes `sage-config` to determine `SAGE_ROOT` and `SAGE_LOCAL`. In the version of `sage-config` supplied by this version of `sage_conf`, we ensure that the symlink from `/var/tmp/sage-....` to the actual install location is set.
 
+Optional steps:
+- Also build the wheelhouse (`local/var/lib/sage/wheels`), at least for tricky packages...
+

Answer 5 · 2021-02-15T06:40:47.000Z

Author: Matthias Koeppe

Answer 6 · 2021-02-15T07:56:12.000Z

comment:4

On macOS, some shared libraries from homebrew still leak into the build:

find src/pkgs/sage_conf-relocatable/.tox/python/lib/python3.9/site-packages/sage_root -name "*.so" -o -name "*.dylib" | xargs otool -L
libgiac.dylib:
/usr/local/opt/gettext/lib/libintl.8.dylib
R/library/cluster/libs/*.dylib:
	/usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version 11.0.0, current version 11.0.0)
	/usr/local/opt/pcre/lib/libpcre.1.dylib (compatibility version 4.0.0, current version 4.12.0)
	/usr/local/opt/xz/lib/liblzma.5.dylib (compatibility version 8.0.0, current version 8.5.0)

Definitely should disable the R build (#30383).

Also check whether setting CC=gcc -sysroot.... helps getting rid of /usr/local leaking in.

Also need to add --enable-fat-binary.

setup.py build_py should probably use a symlink from /var/tmp... into ~/.sage as well so that it is easy to test relocation after removing the symlink.
Also copytree from sage_root_source should be changed so it overwrites (updates) the existing contents of SAGE_ROOT.

the tox test should include the test with otool for self-containedness.

Answer 7 · 2021-02-15T07:56:12.000Z

Description changed:

--- 
+++ 
@@ -2,6 +2,13 @@
 
 The `sage` script invokes `sage-config` to determine `SAGE_ROOT` and `SAGE_LOCAL`. In the version of `sage-config` supplied by this version of `sage_conf`, we ensure that the symlink from `/var/tmp/sage-....` to the actual install location is set.
 
+To test:
+
+```
+(cd src/pkgs/sage_conf-relocatable/ && tox -v -v -v)
+```
+On macOS, this builds a 750MB wheel, an order of magnitude above the standard file size limit on PyPI.
+
 Optional steps:
 - Also build the wheelhouse (`local/var/lib/sage/wheels`), at least for tricky packages...

Answer 8 · 2021-02-16T02:56:32.000Z

Changed commit from 9573b7c to f785d3c

Answer 9 · 2021-02-16T02:56:32.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`f785d3c`	`src/pkgs/sage_conf-relocatable/tox.ini: Add environment macos-10.15`

Answer 10 · 2021-02-16T02:58:06.000Z

comment:6

Using -isysroot helps but giac configuration needs more work - #31403

Answer 11 · 2021-02-16T03:26:34.000Z

comment:7

Replying to @mkoeppe:

setup.py build_py should probably use a symlink from /var/tmp... into ~/.sage as well so that it is easy to test relocation after removing the symlink.

Does not work because sage resolves symlinks when it determines SAGE_ROOT.

Answer 12 · 2021-02-19T00:01:30.000Z

Changed dependencies from #29039 to #29039, #31409

Answer 13 · 2021-02-19T00:02:07.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`6da7ba2`	`Add variable TARGETS`
`b2d4ab8`	`build/pkgs/{r,rpy2}: Downgrade to optional`
`8732076`	`src/sage/repl/ipython_tests.py: Mark R interface tests # optional - rpy2`
`dbdf084`	`src/sage/structure/sage_object.pyx: Mark R interface test # optional - rpy2`
`737b21c`	`src/sage/stats/r.py: Mark all 2 doctests in this file as # optional - rpy2`
`4403924`	`src/sage/interfaces/r.py: Mark all tests # optional - rpy2`
`e56a2a9`	`Merge branch 't/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional' into t/31396/relocatable_wheel_version_of_package_sage_conf`

Answer 14 · 2021-02-19T00:02:07.000Z

Changed commit from f785d3c to e56a2a9

Answer 15 · 2021-02-19T02:19:30.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`a228974`	`src/pkgs/sage_conf-relocatable/setup.py: Move the SAGE_ROOT directory away after building`

Answer 16 · 2021-02-19T02:19:30.000Z

Changed commit from e56a2a9 to a228974

Answer 17 · 2021-02-19T03:35:35.000Z

Changed commit from a228974 to c260942

Answer 18 · 2021-02-19T03:35:35.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`0cac965`	`build/pkgs/ecl: Update to 21.2.1, remove patches included in new version`
`6a62484`	`delete all merged upstream patches: MR 210, MR 231`
`62fac00`	`Merge branch 't/31336/upgrade_ecl_to_21_2_1' into t/31396/relocatable_wheel_version_of_package_sage_conf`
`c260942`	`src/pkgs/sage_conf-relocatable/setup.py: Use --enable-download-from-upstream-url`

Answer 19 · 2021-02-19T06:10:23.000Z

Changed commit from c260942 to 4f17a04

Answer 20 · 2021-02-19T06:10:23.000Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`dc5e225`	`build/pkgs/python3/spkg-configure.m4: On macOS, if the distutils test fails, try using empty ARCHFLAGS`
`df17ad4`	`tox.ini: Add configuration factors for specific homebrew python3.x`
`914ea58`	`build/pkgs/python3/spkg-configure.m4: Get rid of stray AC_SUBST`
`986739b`	`m4/sage_check_python_for_venv.m4: Fix reason`
`0b3e70d`	`build/pkgs/python3/spkg-configure.m4: If PYTHON_FOR_VENV is configured to build multiarch extensions, set SAGE_ARCHFLAGS to disable it`
`372adcb`	`.github/workflows/tox.yml: Update xcode versions`
`fc8b676`	`SAGE_CHECK_PYTHON_FOR_VENV: Rework with less nesting`
`5ae8044`	`Merge branch 'u/mkoeppe/accept__usr_bin_python3_from_xcode_12_3_on_macos_10_15__catalina_' of git://trac.sagemath.org/sage into t/31396/relocatable_wheel_version_of_package_sage_conf`
`c5c73a9`	`src/pkgs/sage_conf-relocatable/tox.ini: Define environments macos-python3.8, macos-python3.7, macos-python3_xcode`
`4f17a04`	`src/pkgs/sage_conf-relocatable/setup.py: Use --enable-fat-binary`

Answer 21 · 2021-02-19T06:13:07.000Z

Description changed:

--- 
+++ 
@@ -5,9 +5,14 @@
 To test:
 
 ```
-(cd src/pkgs/sage_conf-relocatable/ && tox -v -v -v)
+./bootstrap && (cd src/pkgs/sage_conf-relocatable/ && tox -v -v -v)
 ```
-On macOS, this builds a 750MB wheel, an order of magnitude above the standard file size limit on PyPI.
+Specifically for building a wheel on macOS that uses XCode python 3.8:
+
+```
+./bootstrap && (cd src/pkgs/sage_conf-relocatable/ && tox -v -v -v -e python-macos-10.15-python3.8)
+```
+This builds a 750MB wheel, an order of magnitude above the standard file size limit on PyPI.
 
 Optional steps:
 - Also build the wheelhouse (`local/var/lib/sage/wheels`), at least for tricky packages...

Answer 22 · 2021-02-19T06:40:05.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`8514352`	`src/pkgs/sage_conf-relocatable/tox.ini: Run delocate-listdeps`

Answer 23 · 2021-02-19T06:40:05.000Z

Changed commit from 4f17a04 to 8514352

Answer 24 · 2021-02-20T05:07:08.000Z

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making `SAGE_LOCAL` relocatable using the method proposed in #31076, using `SAGE_LOCAL=/var/tmp/sage-...` and a symlink.
+This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making `SAGE_ROOT` (and thus `SAGE_LOCAL=$SAGE_ROOT/local`) relocatable using the method proposed in #31076, using `SAGE_ROOT=/var/tmp/sage-...` and a symlink.
 
 The `sage` script invokes `sage-config` to determine `SAGE_ROOT` and `SAGE_LOCAL`. In the version of `sage-config` supplied by this version of `sage_conf`, we ensure that the symlink from `/var/tmp/sage-....` to the actual install location is set.

Answer 25 · 2021-03-15T06:02:00.000Z

Changed commit from 8514352 to 1baf390

Answer 26 · 2021-03-15T06:02:00.000Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`c3e4093`	`Rename SAGE_OPTIONAL_CLEANED_PACKAGES to SAGE_OPTIONAL_UNINSTALLED_PACKAGES`
`4916415`	`Merge tag '9.3.beta9' into t/30383/new_package_type__optional_enabled_by_default`
`a35428e`	`Merge #30383`
`e098133`	`src/pkgs/sage_conf/sage_conf.py.in: Add SAGE_SPKG_WHEELS`
`5dbd712`	`src/pkgs/sage_conf-pypi/setup.py: Use 'configure --disable-notebook', run 'make build' so that wheels are built`
`839c534`	`src/pkgs/sage_conf-pypi/setup.py: Fix typo`
`100ff51`	`src/pkgs/sage_conf-pypi/MANIFEST.in: Graft sage_root/build/pkgs/sage_docbuild/src`
`afa2ffd`	`Add configure option --disable-sagelib`
`a4f75b8`	`src/pkgs/sage_conf-pypi: Use configure --disable-sagelib, update README`
`1baf390`	`Merge branch 't/29039/pip_installable_package_sage_bootstrap' into t/31396/relocatable_wheel_version_of_package_sage_conf`

Answer 27 · 2021-03-24T17:32:21.000Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`f04c134`	`Merge branch 't/30383/new_package_type__optional_enabled_by_default' into t/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional`
`27c4bb9`	`Merge #31409`
`66c2987`	`build/pkgs/{sage_sws2rst,sage_docbuild}/install-requires.txt: New`
`7f8ec6c`	`build/pkgs/sagelib/src/tox.ini: Add factor sitepackages`
`0283da5`	`build/make/Makefile.in: Add wheel, setuptools_wheel to PYTHON_TOOLCHAIN to make sure that PEP 517 packages have a complete build system`
`f720722`	`build/pkgs/sagelib/src/tox.ini: Add factor nobuildisolation`
`c451b31`	`src/setup.cfg.m4 (install_requires): Add sage_conf`
`6700223`	`Merge tag '9.3.rc0' into t/30913/sagelib__add_setup_cfg__install_requires_`
`ac0f563`	`Merge branch 't/30913/sagelib__add_setup_cfg__install_requires_' into t/29039/pip_installable_package_sage_bootstrap`
`31bba97`	`Merge #29039`

Answer 28 · 2021-03-24T17:32:21.000Z

Changed commit from 1baf390 to 31bba97

Answer 29 · 2021-03-24T17:43:52.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`f5f4a15`	`src/pkgs/sage_conf-relocatable/setup.py: Build the wheelhouse, but --disable-notebook --disable-r --disable-sagelib`

Answer 30 · 2021-03-24T17:43:52.000Z

Changed commit from 31bba97 to f5f4a15

Answer 31 · 2021-03-24T17:51:07.000Z

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making `SAGE_ROOT` (and thus `SAGE_LOCAL=$SAGE_ROOT/local`) relocatable using the method proposed in #31076, using `SAGE_ROOT=/var/tmp/sage-...` and a symlink.
+This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution (and the wheelhouse `local/var/lib/sage/wheels`), making `SAGE_ROOT` (and thus `SAGE_LOCAL=$SAGE_ROOT/local`) relocatable using the method proposed in #31076, using `SAGE_ROOT=/var/tmp/sage-...` and a symlink.
 
 The `sage` script invokes `sage-config` to determine `SAGE_ROOT` and `SAGE_LOCAL`. In the version of `sage-config` supplied by this version of `sage_conf`, we ensure that the symlink from `/var/tmp/sage-....` to the actual install location is set.
 
@@ -14,6 +14,4 @@
 ```
 This builds a 750MB wheel, an order of magnitude above the standard file size limit on PyPI.
 
-Optional steps:
-- Also build the wheelhouse (`local/var/lib/sage/wheels`), at least for tricky packages...

Answer 32 · 2021-03-24T17:52:55.000Z

Description changed:

--- 
+++ 
@@ -14,4 +14,4 @@
 ```
 This builds a 750MB wheel, an order of magnitude above the standard file size limit on PyPI.
 
-
+Follow-up step: Do not package up the wheel-building venv.

Answer 33 · 2021-03-24T19:14:25.000Z

Changed commit from f5f4a15 to 87db1d0

Answer 34 · 2021-03-24T19:14:25.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`87db1d0`	`src/pkgs/sage_conf-relocatable/setup.py: Use --with-mp=gmp instead of broken --without-system-gmp option`

Answer 35 · 2021-03-24T20:33:59.000Z

Changed dependencies from #29039, #31409 to #29039, #31409, #31521

Answer 36 · 2021-03-24T20:35:03.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`128e8ef`	`Revert "do not allow numpy intrinsics when building fat binary"`
`b871e09`	`Merge #31521`

Answer 37 · 2021-03-24T20:35:03.000Z

Changed commit from 87db1d0 to b871e09

Answer 38 · 2021-03-25T00:36:23.000Z

Description changed:

--- 
+++ 
@@ -12,6 +12,6 @@
 ```
 ./bootstrap && (cd src/pkgs/sage_conf-relocatable/ && tox -v -v -v -e python-macos-10.15-python3.8)
 ```
-This builds a 750MB wheel, an order of magnitude above the standard file size limit on PyPI.
+This builds a 900MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985
 
 Follow-up step: Do not package up the wheel-building venv.

Answer 39 · 2021-03-25T01:04:34.000Z

comment:26

delocate-listdeps reports the following library dependencies:

/usr/local/Cellar/gettext/0.21/lib/libintl.8.dylib
/usr/local/Cellar/gmp/6.2.1/lib/libgmp.10.dylib
/usr/local/Cellar/harfbuzz/2.8.0/lib/libharfbuzz.0.dylib
/usr/local/Cellar/libtiff/4.2.0/lib/libtiff.5.dylib
/usr/local/Cellar/libxcb/1.14_1/lib/libxcb.1.1.0.dylib
/usr/local/Cellar/little-cms2/2.12/lib/liblcms2.2.dylib
/usr/local/Cellar/mpfr/4.1.0/lib/libmpfr.6.dylib
/usr/local/Cellar/openjpeg/2.4.0/lib/libopenjp2.2.4.0.dylib
/usr/local/Cellar/readline/8.1/lib/libreadline.8.1.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebp.7.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpdemux.2.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpmux.3.dylib
/usr/local/Cellar/zeromq/4.3.3/lib/libzmq.5.dylib
/usr/local/Cellar/zlib/1.2.11/lib/libz.1.2.11.dylib
@executable_path/../../../../Python3
liba.dylib
libb.dylib
libc.dylib

Answer 40 · 2021-03-25T01:19:42.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`8b5b273`	`build/pkgs/singular/checksums.ini: Use 4.2.0p1+2021-03-24+sage-2`
`f71b013`	`Merge #31552`
`74a0ae0`	`src/pkgs/sage_conf-relocatable/setup.py: Do not package up $SAGE_LOCAL/var/tmp`

Answer 41 · 2021-03-25T01:19:42.000Z

Changed commit from b871e09 to 74a0ae0

Answer 42 · 2021-03-25T05:07:04.000Z

Changed commit from 74a0ae0 to 5eaaa87

Answer 43 · 2021-03-25T05:07:04.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`aa8ff50`	`src/pkgs/sage_conf-relocatable/setup.py: Use --without-system-readline`
`5574aa3`	`build/pkgs/giac/spkg-install.in [macOS]: Pass --with-included-gettext so that it does not link to Homebrew libintl`
`f71ba56`	`configure.ac: If --disable-notebook and --disable-sagelib, do not build zeromq etc.`
`5eaaa87`	`src/pkgs/sage_conf-relocatable/setup.py: Configure libgd without freetype`

Answer 44 · 2021-03-25T06:49:26.000Z

comment:29

/usr/local/Cellar/gettext/0.21/lib/libintl.8.dylib
/usr/local/Cellar/harfbuzz/2.8.0/lib/libharfbuzz.0.dylib
/usr/local/Cellar/libtiff/4.2.0/lib/libtiff.5.dylib
/usr/local/Cellar/libxcb/1.14_1/lib/libxcb.1.1.0.dylib
/usr/local/Cellar/little-cms2/2.12/lib/liblcms2.2.dylib
/usr/local/Cellar/openjpeg/2.4.0/lib/libopenjp2.2.4.0.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebp.7.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpdemux.2.dylib
/usr/local/Cellar/webp/1.2.0/lib/libwebpmux.3.dylib
/usr/local/Cellar/zlib/1.2.11/lib/libz.1.2.11.dylib
@executable_path/../../../../Python3
liba.dylib
libb.dylib
libc.dylib

Answer 45 · 2021-03-25T07:17:24.000Z

comment:30

libintl is still coming in through giac

Answer 46 · 2021-03-25T18:10:07.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`0ccba0c`	`configure.ac (--disable-notebook): Disable argon2_cffi`
`b7051a9`	`configure.ac (--disable-notebook --disable-sagelib): Disable ipython`

Answer 47 · 2021-03-25T18:10:07.000Z

Changed commit from 5eaaa87 to b7051a9

Answer 48 · 2021-03-25T23:36:35.000Z

comment:32

The remaining shared libraries from /usr/local come in through our freetype (which uses libharfbuzz from /usr/local), libgd (which uses libtiff), and pillow (libtiff and everything else).

We could try not to ship matplotlib at all in the wheelhouse - it is responsible for pulling in freetype and pillow - but it still shares one shared library with some of our packages - libpng.

Answer 49 · 2021-03-26T01:51:35.000Z

Changed commit from b7051a9 to 41ac551

Answer 50 · 2021-03-26T01:51:35.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`760b6f0`	`configure.ac: If --disable-notebook and --disable-sagelib, do not build zeromq etc.`
`0df51e5`	`configure.ac (--disable-notebook): Disable argon2_cffi`
`42db661`	`configure.ac (--disable-notebook --disable-sagelib): Disable ipython`
`c63ccb0`	`src/pkgs/sage_conf-relocatable/tox.ini (macos): Also use -isysroot with g++`
`1d12382`	`src/pkgs/sage_conf-relocatable/setup.py: Use LIBGD_CONFIGURE='--without-...almost everything...'`
`1ef46d0`	`build/pkgs/giac: Add another patch in patches/autotools`
`1b0a9ee`	`build/pkgs/giac: Update to 1.6.0.47p3`
`41ac551`	`Merge #31562`

Answer 51 · 2021-03-26T01:55:27.000Z

Changed dependencies from #29039, #31409, #31521 to #29039, #31409, #31521, #31552, #31562

Answer 52 · 2021-03-26T02:03:15.000Z

Changed commit from 41ac551 to 00e3956

Answer 53 · 2021-03-26T02:03:15.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`89f61d2`	`build/pkgs/giac: Update to 1.6.0.47p3`
`00e3956`	`Merge #31562`

Answer 54 · 2021-03-26T02:29:33.000Z

comment:36

Replying to @mkoeppe:

libintl is still coming in through giac

Solved now via #31562.

Answer 55 · 2021-03-26T19:36:08.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`e41932a`	`src/pkgs/sage_conf-relocatable/tox.ini (macos): Add more SDK paths`

Answer 56 · 2021-03-26T19:36:08.000Z

Changed commit from 00e3956 to e41932a

Answer 57 · 2021-03-27T02:46:41.000Z

Work Issues: Refactor through #31567

Answer 58 · 2021-03-27T04:06:04.000Z

comment:39

Replying to @mkoeppe:

The remaining shared libraries from /usr/local come in through our freetype (which uses libharfbuzz from /usr/local)

freetype checks for harfbuzz using pkg-config, accepts configure --without-harfbuzz. https://github.com/freetype/freetype/blob/f9f6adb625c48ef15b5d61a3ac1709a068ea95a3/builds/unix/configure.raw#L504

Answer 59 · 2021-03-27T04:17:22.000Z

Changed dependencies from #29039, #31409, #31521, #31552, #31562 to #29039, #31409, #31521, #31552, #31562, #31567

Answer 60 · 2021-03-27T05:01:53.000Z

Description changed:

--- 
+++ 
@@ -10,8 +10,10 @@
 Specifically for building a wheel on macOS that uses XCode python 3.8:
 
 ```
-./bootstrap && (cd src/pkgs/sage_conf-relocatable/ && tox -v -v -v -e python-macos-10.15-python3.8)
+$ tox -e local-macos-local-macos-10.15-nohomebrew-python3_xcode -- bash
+[tox -e ...] $ (cd src/pkgs/sage_conf-relocatable/ && /usr/local/bin/tox -v -v -v)
 ```
 This builds a 900MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985
 
 Follow-up step: Do not package up the wheel-building venv.
+

Answer 61 · 2021-03-27T05:08:34.000Z

Changed commit from e41932a to 4ed1bf4

Answer 62 · 2021-03-27T05:08:34.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`87e2dba`	`tox.ini: Add local-macos-nohomebrew and configuration facors macos-{10.14,10.15,11.1}`
`0458f23`	`tox.ini: Add macos-{python3_xcode,nohomebrew}-python{3.7,3.8}`
`39629dd`	`Merge #31552`
`9d9c562`	`Merge #31562`
`a5f1fc7`	`Merge #31567`
`3dd4885`	`build/pkgs/pplpy/dependencies: Depend on sphinx only if SAGE_SPKG_INSTALL_DOCS!=no`
`68c447f`	`tox.ini (local -- bash): Set PS1`
`4ed1bf4`	`src/pkgs/sage_conf-relocatable: Remove settings that are now set in SAGE_ROOT/tox.ini`

Answer 63 · 2021-03-27T05:09:36.000Z

Changed work issues from Refactor through #31567 to none

Answer 64 · 2021-03-27T05:56:02.000Z

Description changed:

--- 
+++ 
@@ -11,7 +11,7 @@
 
 ```
 $ tox -e local-macos-local-macos-10.15-nohomebrew-python3_xcode -- bash
-[tox -e ...] $ (cd src/pkgs/sage_conf-relocatable/ && /usr/local/bin/tox -v -v -v)
+[tox -e ...] $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel)
 ```
 This builds a 900MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985

Answer 65 · 2021-03-28T02:23:18.000Z

Description changed:

--- 
+++ 
@@ -10,8 +10,8 @@
 Specifically for building a wheel on macOS that uses XCode python 3.8:
 
 ```
-$ tox -e local-macos-local-macos-10.15-nohomebrew-python3_xcode -- bash
-[tox -e ...] $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel)
+$ tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
+(tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel)
 ```
 This builds a 900MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985

Answer 66 · 2021-03-28T04:00:51.000Z

Description changed:

--- 
+++ 
@@ -10,7 +10,7 @@
 Specifically for building a wheel on macOS that uses XCode python 3.8:
 
 ```
-$ tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
+$ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
 (tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel)
 ```
 This builds a 900MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985

Answer 67 · 2021-03-28T06:43:49.000Z

Changed commit from 4ed1bf4 to e4270dd

Answer 68 · 2021-03-28T06:43:49.000Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`300d054`	`build/pkgs/pillow/spkg-install.in: Rename environment variable extra_build_ext to PILLOW_BUILD_EXT`
`8c8d83d`	`build/pkgs/freetype/spkg-install.in: Pass environment variable FREETYPE_CONFIGURE to configure`
`12c84a3`	`tox.ini (macos-nohomebrew): Prevent /usr/local from leaking in through boost, freetype, pillow`
`7966b66`	`Merge #31409`
`fadd2b9`	`Merge #31567`
`59f41b9`	`src/pkgs/sage_conf-relocatable/setup.py: Resolve SAGE_ROOT`
`b5edde2`	`tox.ini (macos-nohomebrew): Fix typo in PILLOW_...`
`e15b152`	`Merge #31567`
`dffcdbc`	`tox.ini (macos-nohomebrew): Fix up pillow build by setting ZLIB_ROOT`
`e4270dd`	`Merge #31567`

Answer 69 · 2021-03-29T02:00:22.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`4069297`	`src/pkgs/sage_conf-relocatable/setup.py: Unlink a previous symlink 'sage_root' in the package source tree before setting the new one`
`3779a25`	`tox.ini (macos-nohomebrew): Disable more libraries to remove dependencies on homebrew`
`587142f`	`Merge #31567`

Answer 70 · 2021-03-29T02:00:22.000Z

Changed commit from e4270dd to 587142f

Answer 71 · 2021-03-29T04:44:58.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`c5a823c`	`src/pkgs/sage_conf-relocatable/setup.py: Make sure that we do not package a copy of the sagelib source tree`

Answer 72 · 2021-03-29T04:44:58.000Z

Changed commit from 587142f to c5a823c

Answer 73 · 2021-03-29T04:46:15.000Z

Description changed:

--- 
+++ 
@@ -13,7 +13,7 @@
 $ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
 (tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel)
 ```
-This builds a 900MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985
+This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985
 
 Follow-up step: Do not package up the wheel-building venv.

Answer 74 · 2021-03-29T06:10:57.000Z

Description changed:

--- 
+++ 
@@ -15,5 +15,5 @@
 ```
 This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985
 
-Follow-up step: Do not package up the wheel-building venv.
+Follow-up step: Do not package up the wheel-building venv (247MB in `sage_root/local/lib/python3.8/site-packages/`); disable all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible

Answer 75 · 2021-03-29T06:31:15.000Z

Description changed:

--- 
+++ 
@@ -15,5 +15,5 @@
 ```
 This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985
 
-Follow-up step: Do not package up the wheel-building venv (247MB in `sage_root/local/lib/python3.8/site-packages/`); disable all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible
+**Follow-up steps:** Do not package up the wheel-building venv (247MB in `sage_root/local/lib/python3.8/site-packages/`); do not ship any `any` wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible

Answer 76 · 2021-03-29T17:09:10.000Z

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution (and the wheelhouse `local/var/lib/sage/wheels`), making `SAGE_ROOT` (and thus `SAGE_LOCAL=$SAGE_ROOT/local`) relocatable using the method proposed in #31076, using `SAGE_ROOT=/var/tmp/sage-...` and a symlink.
+This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution (and the wheelhouse `local/var/lib/sage/wheels`), making `SAGE_ROOT` (and thus `SAGE_LOCAL=$SAGE_ROOT/local`) relocatable using Marc Culler's `/var/tmp` surgery (the method proposed in #31076, using `SAGE_ROOT=/var/tmp/sage-...` and a symlink).
 
 The `sage` script invokes `sage-config` to determine `SAGE_ROOT` and `SAGE_LOCAL`. In the version of `sage-config` supplied by this version of `sage_conf`, we ensure that the symlink from `/var/tmp/sage-....` to the actual install location is set.
 
@@ -15,5 +15,14 @@
 ```
 This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985
 
+In the meantime, you can use a prebuilt wheel (so far, macOS python3.8 x86_64 only) made available at https://github.com/mkoeppe/sage/releases;
+install the wheel using
+
+```
+pip install https://github.com/mkoeppe/sage/releases/download/9.3.rc0%2Btrac31396-2021-03-28/sage_conf-9.3rc0-cp38-cp38-macosx_10_14_6_x86_64.whl
+```
+Then proceed with the same instructions as https://pypi.org/project/sage-conf/
+
+
 **Follow-up steps:** Do not package up the wheel-building venv (247MB in `sage_root/local/lib/python3.8/site-packages/`); do not ship any `any` wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible

Answer 77 · 2021-03-29T17:12:50.000Z

Description changed:

--- 
+++ 
@@ -19,7 +19,7 @@
 install the wheel using
 
 ```
-pip install https://github.com/mkoeppe/sage/releases/download/9.3.rc0%2Btrac31396-2021-03-28/sage_conf-9.3rc0-cp38-cp38-macosx_10_14_6_x86_64.whl
+/usr/bin/python3 -m pip install --user https://github.com/mkoeppe/sage/releases/download/9.3.rc0%2Btrac31396-2021-03-28/sage_conf-9.3rc0-cp38-cp38-macosx_10_14_6_x86_64.whl
 ```
 Then proceed with the same instructions as https://pypi.org/project/sage-conf/

Answer 78 · 2021-03-30T16:48:42.000Z

Description changed:

--- 
+++ 
@@ -24,5 +24,7 @@
 Then proceed with the same instructions as https://pypi.org/project/sage-conf/
 
 
-**Follow-up steps:** Do not package up the wheel-building venv (247MB in `sage_root/local/lib/python3.8/site-packages/`); do not ship any `any` wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible
+**Follow-up steps:** 
+- Do not package up the wheel-building venv (247MB in `sage_root/local/lib/python3.8/site-packages/`); do not ship any `any` wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
+- split out `sage_local` from `sage_conf` and use install-requires in the `sage_conf` wheels with @ links to GH releases to the `sage_local` wheel and the built non-`any` wheels.

Answer 79 · 2021-03-31T01:19:50.000Z

Description changed:

--- 
+++ 
@@ -25,6 +25,6 @@
 
 
 **Follow-up steps:** 
-- Do not package up the wheel-building venv (247MB in `sage_root/local/lib/python3.8/site-packages/`); do not ship any `any` wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
+- Do not package up the wheel-building venv (247MB in `sage_root/local/lib/python3.8/site-packages/`) (#29013); do not ship any `any` wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
 - split out `sage_local` from `sage_conf` and use install-requires in the `sage_conf` wheels with @ links to GH releases to the `sage_local` wheel and the built non-`any` wheels.

Answer 80 · 2021-04-03T16:50:27.000Z

Description changed:

--- 
+++ 
@@ -13,7 +13,7 @@
 $ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
 (tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel)
 ```
-This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI. Opened a PR to request a file size limit increase: https://github.com/pypa/pypi-support/issues/985
+This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI; a file size limit increase (requested in  https://github.com/pypa/pypi-support/issues/985) has been granted by the PyPI team.
 
 In the meantime, you can use a prebuilt wheel (so far, macOS python3.8 x86_64 only) made available at https://github.com/mkoeppe/sage/releases;
 install the wheel using
@@ -26,5 +26,5 @@
 
 **Follow-up steps:** 
 - Do not package up the wheel-building venv (247MB in `sage_root/local/lib/python3.8/site-packages/`) (#29013); do not ship any `any` wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
-- split out `sage_local` from `sage_conf` and use install-requires in the `sage_conf` wheels with @ links to GH releases to the `sage_local` wheel and the built non-`any` wheels.
+- #31602: split out `sage_root` from `sage_conf` and use install-requires in the `sage_conf` sdist/wheel with @ links to GH releases to the `sage_root` wheel and the built non-`any` wheels.

Answer 81 · 2021-04-03T20:50:58.000Z

Changed dependencies from #29039, #31409, #31521, #31552, #31562, #31567 to #29039, #31409, #31521, #31552, #31562, #31567, #29013

Answer 82 · 2021-04-03T21:07:17.000Z

Description changed:

--- 
+++ 
@@ -11,7 +11,7 @@
 
 ```
 $ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
-(tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel)
+(tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel --plat-name macosx_10_15_x86_64)
 ```
 This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI; a file size limit increase (requested in  https://github.com/pypa/pypi-support/issues/985) has been granted by the PyPI team.

Answer 83 · 2021-04-03T21:08:50.000Z

Changed commit from c5a823c to a3ccb60

Answer 84 · 2021-04-03T21:08:50.000Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`30926d8`	`.github/workflows/tox*.yml: Replace ubuntu-focal-i386 by manylinux-2_24-i686`
`a06ccf7`	`Merge #31541`
`bf6e5f8`	`build/make/Makefile.in: Change some SAGE_LOCAL to SAGE_VENV`
`832f96a`	`build/make/Makefile.in: Rework using tree_... variables`
`3436ad6`	`build/make/Makefile.in: Define SAGE_VENV earlier`
`8cc0b3c`	`WIP configure --with-sage-venv`
`1cfbe2b`	`build/bin/sage-spkg: Change more SAGE_LOCAL to SAGE_INST_LOCAL`
`2202f84`	`configure.ac: Finish --with-sage-venv handling`
`fa897c6`	`Merge branch 't/29013/separate_venvs' into t/31396/relocatable_wheel_version_of_package_sage_conf`
`a3ccb60`	`src/pkgs/sage_conf-relocatable/setup.py: Build the wheels in a venv outside of SAGE_LOCAL`

Answer 85 · 2021-04-04T00:49:29.000Z

Changed commit from a3ccb60 to a295b81

Answer 86 · 2021-04-04T00:49:29.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`b604a68`	`build/make/Makefile.in: Also source sage-src-env-config for script packages`
`92f1a1b`	`build/pkgs/pynac/spkg-install.in: Build with PYTHON_FOR_VENV`
`c94af4b`	`Merge branch 't/29013/separate_venvs' into t/31396/relocatable_wheel_version_of_package_sage_conf`
`9f1f794`	`src/pkgs/sage_conf-relocatable/setup.py: Include plat_name in the directory name for SAGE_ROOT`
`d19c5a4`	`Fixup`
`5382ec7`	`src/pkgs/sage_conf-relocatable/setup.py: Working version`
`a295b81`	`.gitignore: Add more`

Answer 87 · 2021-04-04T02:15:57.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`e4e6a1e`	`src/pkgs/sage_conf-relocatable/setup.py: Fixup`
`323dcf1`	`src/pkgs/sage_conf-relocatable/setup.py: Add first install_requires @ GH`
`4049273`	`src/pkgs/sage_conf-relocatable/setup.py: Add install_requires for all built binary wheels`

Answer 88 · 2021-04-04T02:15:57.000Z

Changed commit from a295b81 to 4049273

Answer 89 · 2021-04-04T02:37:06.000Z

Description changed:

--- 
+++ 
@@ -1,30 +1,33 @@
-This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution (and the wheelhouse `local/var/lib/sage/wheels`), making `SAGE_ROOT` (and thus `SAGE_LOCAL=$SAGE_ROOT/local`) relocatable using Marc Culler's `/var/tmp` surgery (the method proposed in #31076, using `SAGE_ROOT=/var/tmp/sage-...` and a symlink).
+This version of `sage_conf` is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making `SAGE_ROOT` (and thus `SAGE_LOCAL=$SAGE_ROOT/local`) relocatable using Marc Culler's symbolic link surgery (the method proposed in #31076, using `SAGE_ROOT=/var/tmp/sage-...` and a symlink).
 
 The `sage` script invokes `sage-config` to determine `SAGE_ROOT` and `SAGE_LOCAL`. In the version of `sage-config` supplied by this version of `sage_conf`, we ensure that the symlink from `/var/tmp/sage-....` to the actual install location is set.
 
-To test:
+So far, a wheel has been built for XCode python 3.8 on macOS 10.15.  The wheel is 670MB in size, an order of magnitude above the standard file size limit on PyPI; but a file size limit increase (requested in  https://github.com/pypa/pypi-support/issues/985) has kindly been granted by the PyPI team.
+
+The wheel declares dependencies (`install-requires`) to all Python packages in the Sage distribution that have extension modules.  The dependencies are specific using `@` to URLs on 
+https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1, where I have uploaded the binary wheels.
+
+**Instructions for testing:**
 
 ```
-./bootstrap && (cd src/pkgs/sage_conf-relocatable/ && tox -v -v -v)
-```
-Specifically for building a wheel on macOS that uses XCode python 3.8:
+$ /usr/bin/python3 -m pip install -U --user sage-conf
+``` 
+
+
+
+**Procedure for building the sage_conf wheel and compatible wheels:**
+
+
+For building a wheel on macOS that uses XCode python 3.8:
 
 ```
 $ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
 (tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel --plat-name macosx_10_15_x86_64)
 ```
-This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI; a file size limit increase (requested in  https://github.com/pypa/pypi-support/issues/985) has been granted by the PyPI team.
 
-In the meantime, you can use a prebuilt wheel (so far, macOS python3.8 x86_64 only) made available at https://github.com/mkoeppe/sage/releases;
-install the wheel using
-
-```
-/usr/bin/python3 -m pip install --user https://github.com/mkoeppe/sage/releases/download/9.3.rc0%2Btrac31396-2021-03-28/sage_conf-9.3rc0-cp38-cp38-macosx_10_14_6_x86_64.whl
-```
-Then proceed with the same instructions as https://pypi.org/project/sage-conf/
 
 
 **Follow-up steps:** 
-- Do not package up the wheel-building venv (247MB in `sage_root/local/lib/python3.8/site-packages/`) (#29013); do not ship any `any` wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
+- do not ship the Cython wheel; disable building all sphinx packages; build sagelib too; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
 - #31602: split out `sage_root` from `sage_conf` and use install-requires in the `sage_conf` sdist/wheel with @ links to GH releases to the `sage_root` wheel and the built non-`any` wheels.

Answer 90 · 2021-04-04T02:47:06.000Z

Description changed:

--- 
+++ 
@@ -10,7 +10,7 @@
 **Instructions for testing:**
 
 ```
-$ /usr/bin/python3 -m pip install -U --user sage-conf
+$ /usr/bin/python3 -m pip install -U --user --only-binary sage-conf sage-conf
 ```

Answer 91 · 2021-04-04T04:53:39.000Z

comment:69

Unfortunately it turns out that PyPI does not allow @ references in install requires (see pypa/pip#6301)

HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
Invalid value for requires_dist. Error: Can't have direct dependency: 'Cython @ https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/Cython-0.29.21-cp38-cp38-macosx_10_15_x86_64.whl'

Answer 92 · 2021-04-04T05:16:38.000Z

Description changed:

--- 
+++ 
@@ -10,7 +10,7 @@
 **Instructions for testing:**
 
 ```
-$ /usr/bin/python3 -m pip install -U --user --only-binary sage-conf sage-conf
+$ /usr/bin/python3 -m pip install -U --user --only-binary sage-conf https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sage_conf-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl
 ```

Answer 93 · 2021-04-04T05:18:59.000Z

Description changed:

--- 
+++ 
@@ -2,10 +2,11 @@
 
 The `sage` script invokes `sage-config` to determine `SAGE_ROOT` and `SAGE_LOCAL`. In the version of `sage-config` supplied by this version of `sage_conf`, we ensure that the symlink from `/var/tmp/sage-....` to the actual install location is set.
 
-So far, a wheel has been built for XCode python 3.8 on macOS 10.15.  The wheel is 670MB in size, an order of magnitude above the standard file size limit on PyPI; but a file size limit increase (requested in  https://github.com/pypa/pypi-support/issues/985) has kindly been granted by the PyPI team.
+So far, a wheel has been built for XCode python 3.8 on macOS 10.15.  The wheel is 670MB in size, an order of magnitude above the standard file size limit on PyPI; but a file size limit increase (requested in  https://github.com/pypa/pypi-support/issues/985) has kindly been granted by the PyPI team.  
 
 The wheel declares dependencies (`install-requires`) to all Python packages in the Sage distribution that have extension modules.  The dependencies are specific using `@` to URLs on 
-https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1, where I have uploaded the binary wheels.
+https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1, where I have uploaded the binary wheels.  Unfortunately, by PyPI policy, such `@` references are not allowed for packages on PyPI. 
+Hence, the pip invocation needs to use a URL.
 
 **Instructions for testing:**

Answer 94 · 2021-04-04T21:15:41.000Z

Description changed:

--- 
+++ 
@@ -14,17 +14,19 @@
 $ /usr/bin/python3 -m pip install -U --user --only-binary sage-conf https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sage_conf-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl
 ``` 
 
-
-
 **Procedure for building the sage_conf wheel and compatible wheels:**
 
 
-For building a wheel on macOS that uses XCode python 3.8:
+For building wheels on macOS that uses XCode python 3.8:
 
 ```
 $ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
 (tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel --plat-name macosx_10_15_x86_64)
 ```
+
+This creates:
+- src/pkgs/sage_conf-relocatable/dist/sage_conf-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl
+- src/pkgs/sage_conf-relocatable/sage_root/venv-cpython-38-darwin-macosx_10_15_x86_64/var/lib/sage/wheels/*.whl

Answer 95 · 2021-04-04T21:26:48.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`916bda8`	`configure.ac: Add configure option --disable-doc`
`5871e7c`	`Have sagemath_standard wheel depend on sage_conf wheel @ GH`

Answer 96 · 2021-04-04T21:26:48.000Z

Changed commit from 4049273 to 5871e7c

Answer 97 · 2021-04-04T23:49:25.000Z

Description changed:

--- 
+++ 
@@ -11,7 +11,9 @@
 **Instructions for testing:**
 
 ```
-$ /usr/bin/python3 -m pip install -U --user --only-binary sage-conf https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sage_conf-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl
+$ export PATH=$HOME/Library/Python/3.8/bin:/usr/bin:/bin:/usr/sbin:/sbin
+$ /usr/bin/python3 -m pip install -U --user  https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sagemath_standard-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl
+$ sage
 ``` 
 
 **Procedure for building the sage_conf wheel and compatible wheels:**
@@ -29,7 +31,6 @@
 - src/pkgs/sage_conf-relocatable/sage_root/venv-cpython-38-darwin-macosx_10_15_x86_64/var/lib/sage/wheels/*.whl
 
 
-
 **Follow-up steps:** 
 - do not ship the Cython wheel; disable building all sphinx packages; build sagelib too; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
 - #31602: split out `sage_root` from `sage_conf` and use install-requires in the `sage_conf` sdist/wheel with @ links to GH releases to the `sage_root` wheel and the built non-`any` wheels.

Answer 98 · 2021-04-04T23:50:01.000Z

Changed commit from 5871e7c to 6a3d380

Answer 99 · 2021-04-04T23:50:01.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`7ff26df`	`git grep -l SAGE_SUITESPARSE \| xargs sed -i .bak s/SAGE_SUITESPARSE_LOCALINSTALL/SAGE_SUITESPARSE_PREFIX/g`
`52537d1`	`build/pkgs/cvxopt/spkg-install.in: Actually use the value of SAGE_SUITESPARSE_PREFIX`
`ed1a089`	`Merge #31584`
`be5b7bc`	`Merge #31567`
`1fb3a9b`	`build/pkgs/pynac/spkg-install.in: Back to PYTHON=python3`
`24ebfb5`	`src/bin/sage: Only do the check for an incomplete installation when run from the source tree`
`f331ee7`	`src/pkgs/sage_conf-relocatable/setup.py: Pass CONFIGURED_{CC,CXX} to configure if set`
`6a3d380`	`src/pkgs/sage_conf-relocatable/setup.py: Package libexec too`

Answer 100 · 2021-04-05T00:05:26.000Z

Description changed:

--- 
+++ 
@@ -32,6 +32,7 @@
 
 
 **Follow-up steps:** 
-- do not ship the Cython wheel; disable building all sphinx packages; build sagelib too; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
+- build wheels for `manylinux` using `tox -e docker-manylinux....`
+- disable building the dependencies of the sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible; 
 - #31602: split out `sage_root` from `sage_conf` and use install-requires in the `sage_conf` sdist/wheel with @ links to GH releases to the `sage_root` wheel and the built non-`any` wheels.