Find and uninstall broken installed SPKGs and wheels

Question

Find and uninstall broken installed SPKGs and wheels

mkoeppe opened this issue 3 years ago · 83 comments

When system packages are upgraded, shared libraries that a Sage installation links to can disappear, rendering the Sage installation broken. This is a common problem, reported for example in https://groups.google.com/g/sage-devel/c/Iz8ZsmQM3Pg/m/CPl9CbHmBgAJ

To avoid a full rebuild (make distclean && make build), we can try to find the specific broken packages.

The new command make -j list-broken-packages assists the user with this:

$ make -j list-broken-packages
make --no-print-directory auditwheel_or_delocate-no-deps
sage-logger -p 'sage --pip install -r "/Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt"' '/Users/mkoeppe/s/sage/sage-rebasing/logs/pkgs/auditwheel_or_delocate.log'
[auditwheel_or_delocate] Ignoring auditwheel: markers 'sys_platform != "darwin"' don't match your environment
[auditwheel_or_delocate] Requirement already satisfied: delocate in /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages (from -r /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (0.10.2)
[auditwheel_or_delocate] Requirement already satisfied: wheel>=0.32.0 in /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages (from delocate->-r /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (0.37.1)
[auditwheel_or_delocate] Requirement already satisfied: typing-extensions in /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages (from delocate->-r /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (3.10.0.0)
make[2]: warning: -jN forced in submake: disabling jobserver mode.
# Checking /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/installed/bliss-0.73+debian-1+sage-2016-08-02.p0
...
Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/lib/libumfpack.dylib'
Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/var/tmp/sage/build/suitesparse-5.10.1/src/lib/libsliplu.1.0.2.dylib'
Error during installcheck of 'suitesparse': /Users/mkoeppe/s/sage/sage-rebasing/local/var/tmp/sage/build/suitesparse-5.10.1/src/lib/libsliplu.1.0.2.dylib
Uninstall broken packages by typing:


    make antic-SAGE_LOCAL-uninstall;
    make e_antic-SAGE_LOCAL-uninstall;
    make lcalc-SAGE_LOCAL-uninstall;
    make ratpoints-SAGE_LOCAL-uninstall;
    make normaliz-SAGE_LOCAL-uninstall;
    make r-SAGE_LOCAL-uninstall;
    make backports_zoneinfo-SAGE_VENV-uninstall;
    make fastjsonschema-SAGE_VENV-uninstall;
    make pytz_deprecation_shim-SAGE_VENV-uninstall;
    make stack_data-SAGE_VENV-uninstall;
    make tinycss2-SAGE_VENV-uninstall;
    make tzdata-SAGE_VENV-uninstall;
    make suitesparse-SAGE_LOCAL-uninstall;

real	2m25.787s
user	4m54.270s
sys	4m0.796s

The core functionality is implemented by the new command sage-spkg-installcheck (implemented in sage_bootstrap):

$ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose givaro'
Checking shared library file 'lib/libgivaro.9.dylib'
Checking shared library file 'lib/libgivaro.dylib'
$ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose matplotlib $SAGE_VENV'
Checking wheel file 'var/lib/sage/wheels/matplotlib-3.5.1-cp310-cp310-macosx_12_0_x86_64.whl'
$ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose e_antic'
Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/lib/libeantic.0.dylib'
ERROR [libsana|get_dependencies:137]: 
/usr/local/opt/flint/lib/libflint-16.dylib not found:
  Needed by: /Users/mkoeppe/s/sage/sage-rebasing/local/lib/libeantic.0.dylib
Error during installcheck of 'e_antic': Could not find all dependencies.

It inspects the installation records in $SAGE_LOCAL/var/lib/sage/installed/ and all shared libraries and platform wheels listed there.

On macOS, instead of using otool directly, we use delocate-listdeps from https://pypi.org/project/delocate/

On Linux, instead of using ldd directly, we use auditwheel.lddtree.lddtree (https://github.com/pypa/auditwheel/blob/main/src/auditwheel/lddtree.py)

(Both packages are also used by cibuildwheel)

See also: #34205 Optionally run auditwheel/delocate on wheels before installing them

Depends on #29097

CC: @dimpase @jhpalmieri @nbruin @EmmanuelCharpentier

Component: build

Author: Matthias Koeppe

Branch/Commit: 3c70a38

Reviewer: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/34203

Answer 1 · 2022-07-20T23:09:51.000Z

Description changed:

--- 
+++ 
@@ -1,3 +1,4 @@
 We can inspect the installation records in `$SAGE_LOCAL/var/lib/sage/installed/` by running `ldd` on any shared libraries and uninstall them if they refer to nonexisting libraries.
 
+The core functionality is implemented by the new command `build/bin/sage-spkg-installcheck` (implemented in sage_bootstrap)

Answer 2 · 2022-07-20T23:21:30.000Z

Description changed:

--- 
+++ 
@@ -2,3 +2,10 @@
 
 The core functionality is implemented by the new command `build/bin/sage-spkg-installcheck` (implemented in sage_bootstrap)
 
+On macOS, instead of using `otool` directly, we use `delocate-listdeps` from https://pypi.org/project/delocate/
+
+On Linux, instead of using `ldd` directly, we use `auditwheel show` from https://pypi.org/project/auditwheel/
+
+(Both packages are also used by cibuildwheel)
+
+

Answer 3 · 2022-07-20T23:35:56.000Z

Description changed:

--- 
+++ 
@@ -1,6 +1,6 @@
 We can inspect the installation records in `$SAGE_LOCAL/var/lib/sage/installed/` by running `ldd` on any shared libraries and uninstall them if they refer to nonexisting libraries.
 
-The core functionality is implemented by the new command `build/bin/sage-spkg-installcheck` (implemented in sage_bootstrap)
+The core functionality is implemented by the new command `sage -package installcheck` (implemented in sage_bootstrap)
 
 On macOS, instead of using `otool` directly, we use `delocate-listdeps` from https://pypi.org/project/delocate/

Answer 4 · 2022-07-20T23:41:04.000Z

Description changed:

--- 
+++ 
@@ -1,6 +1,6 @@
 We can inspect the installation records in `$SAGE_LOCAL/var/lib/sage/installed/` by running `ldd` on any shared libraries and uninstall them if they refer to nonexisting libraries.
 
-The core functionality is implemented by the new command `sage -package installcheck` (implemented in sage_bootstrap)
+The core functionality is implemented by the new command `sage-spkg-installcheck` (implemented in sage_bootstrap)
 
 On macOS, instead of using `otool` directly, we use `delocate-listdeps` from https://pypi.org/project/delocate/

Answer 5 · 2022-07-20T23:55:14.000Z

Branch: u/mkoeppe/find_and_uninstall_broken_installed_spkgs_and_wheels

Answer 6 · 2022-07-21T00:00:40.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`562fcab`	`build/bin/sage-spkg-installcheck, build/sage_bootstrap/installcheck.py: New`

Answer 7 · 2022-07-21T00:00:40.000Z

Commit: 562fcab

Answer 8 · 2022-07-21T00:04:37.000Z

Description changed:

--- 
+++ 
@@ -1,6 +1,14 @@
 We can inspect the installation records in `$SAGE_LOCAL/var/lib/sage/installed/` by running `ldd` on any shared libraries and uninstall them if they refer to nonexisting libraries.
 
-The core functionality is implemented by the new command `sage-spkg-installcheck` (implemented in sage_bootstrap)
+The core functionality is implemented by the new command `sage-spkg-installcheck` (implemented in sage_bootstrap):
+
+```
+$ ./sage -sh -c 'build/bin/sage-spkg-installcheck --verbose givaro'
+Checking shared library file 'lib/libgivaro.9.dylib'
+Checking shared library file 'lib/libgivaro.dylib'
+$ ./sage -sh -c 'build/bin/sage-spkg-installcheck --verbose matplotlib $SAGE_VENV'
+Checking wheel file 'var/lib/sage/wheels/matplotlib-3.5.1-cp310-cp310-macosx_12_0_x86_64.whl'
+```
 
 On macOS, instead of using `otool` directly, we use `delocate-listdeps` from https://pypi.org/project/delocate/

Answer 9 · 2022-07-21T00:21:40.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`5faa8cb`	`build/pkgs/auditwheel_or_delocate: New`

Answer 10 · 2022-07-21T00:21:40.000Z

Changed commit from 562fcab to 5faa8cb

Answer 11 · 2022-07-21T00:35:08.000Z

Description changed:

--- 
+++ 
@@ -3,10 +3,10 @@
 The core functionality is implemented by the new command `sage-spkg-installcheck` (implemented in sage_bootstrap):
 
 ```
-$ ./sage -sh -c 'build/bin/sage-spkg-installcheck --verbose givaro'
+$ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose givaro'
 Checking shared library file 'lib/libgivaro.9.dylib'
 Checking shared library file 'lib/libgivaro.dylib'
-$ ./sage -sh -c 'build/bin/sage-spkg-installcheck --verbose matplotlib $SAGE_VENV'
+$ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose matplotlib $SAGE_VENV'
 Checking wheel file 'var/lib/sage/wheels/matplotlib-3.5.1-cp310-cp310-macosx_12_0_x86_64.whl'
 ```

Answer 12 · 2022-07-21T00:46:33.000Z

Changed commit from 5faa8cb to 9265416

Answer 13 · 2022-07-21T00:46:33.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`9265416`	`build/sage_bootstrap/installcheck.py: Implement for macOS`

Answer 14 · 2022-07-21T00:47:19.000Z

Description changed:

--- 
+++ 
@@ -8,6 +8,12 @@
 Checking shared library file 'lib/libgivaro.dylib'
 $ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose matplotlib $SAGE_VENV'
 Checking wheel file 'var/lib/sage/wheels/matplotlib-3.5.1-cp310-cp310-macosx_12_0_x86_64.whl'
+$ ./sage -sh -c 'python3 build/bin/sage-spkg-installcheck --verbose e_antic'
+Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/lib/libeantic.0.dylib'
+ERROR [libsana|get_dependencies:137]: 
+/usr/local/opt/flint/lib/libflint-16.dylib not found:
+  Needed by: /Users/mkoeppe/s/sage/sage-rebasing/local/lib/libeantic.0.dylib
+Error during installcheck of 'e_antic': Could not find all dependencies.
 ```
 
 On macOS, instead of using `otool` directly, we use `delocate-listdeps` from https://pypi.org/project/delocate/

Answer 15 · 2022-07-21T01:36:55.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`a18b205`	`build/sage_bootstrap/installcheck.py: Do not warn if there are no files to check`
`ad92af4`	`build/sage_bootstrap/installcheck.py: Skip pure Python wheels`
`c4e3371`	`build/make/Makefile.in (list-broken-packages): New`

Answer 16 · 2022-07-21T01:36:55.000Z

Changed commit from 9265416 to c4e3371

Answer 17 · 2022-07-21T01:37:30.000Z

Author: Matthias Koeppe

Answer 18 · 2022-07-21T01:37:30.000Z

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-We can inspect the installation records in `$SAGE_LOCAL/var/lib/sage/installed/` by running `ldd` on any shared libraries and uninstall them if they refer to nonexisting libraries.
+We can inspect the installation records in `$SAGE_LOCAL/var/lib/sage/installed/` by running `ldd`/`otool` on any shared libraries and uninstall them if they refer to nonexisting libraries.
 
 The core functionality is implemented by the new command `sage-spkg-installcheck` (implemented in sage_bootstrap):

Answer 19 · 2022-07-21T01:38:25.000Z

Description changed:

--- 
+++ 
@@ -1,4 +1,6 @@
 We can inspect the installation records in `$SAGE_LOCAL/var/lib/sage/installed/` by running `ldd`/`otool` on any shared libraries and uninstall them if they refer to nonexisting libraries.
+
+The new command `make list-broken-packages` does this.
 
 The core functionality is implemented by the new command `sage-spkg-installcheck` (implemented in sage_bootstrap):

Answer 20 · 2022-07-21T02:00:02.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`dbdf33b`	`build/sage_bootstrap/installcheck.py: Only warn when delocate is not available`
`08388f7`	`build/make/Makefile.in (list-broken-packages): Parallelize it`

Answer 21 · 2022-07-21T02:00:02.000Z

Changed commit from c4e3371 to 08388f7

Answer 22 · 2022-07-21T02:02:18.000Z

Description changed:

--- 
+++ 
@@ -1,6 +1,28 @@
 We can inspect the installation records in `$SAGE_LOCAL/var/lib/sage/installed/` by running `ldd`/`otool` on any shared libraries and uninstall them if they refer to nonexisting libraries.
 
-The new command `make list-broken-packages` does this.
+The new command `make -j list-broken-packages` does this:
+
+```
+$ make -j list-broken-packages
+make --no-print-directory auditwheel_or_delocate-no-deps
+sage-logger -p 'sage --pip install -r "/Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt"' '/Users/mkoeppe/s/sage/sage-rebasing/logs/pkgs/auditwheel_or_delocate.log'
+[auditwheel_or_delocate] Ignoring auditwheel: markers 'sys_platform != "darwin"' don't match your environment
+[auditwheel_or_delocate] Requirement already satisfied: delocate in /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages (from -r /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (0.10.2)
+[auditwheel_or_delocate] Requirement already satisfied: wheel>=0.32.0 in /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages (from delocate->-r /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (0.37.1)
+[auditwheel_or_delocate] Requirement already satisfied: typing-extensions in /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages (from delocate->-r /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/auditwheel_or_delocate/requirements.txt (line 1)) (3.10.0.0)
+make[2]: warning: -jN forced in submake: disabling jobserver mode.
+# Checking /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/installed/bliss-0.73+debian-1+sage-2016-08-02.p0
+...
+Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/lib/libumfpack.dylib'
+Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/var/tmp/sage/build/suitesparse-5.10.1/src/lib/libsliplu.1.0.2.dylib'
+Error during installcheck of 'suitesparse': /Users/mkoeppe/s/sage/sage-rebasing/local/var/tmp/sage/build/suitesparse-5.10.1/src/lib/libsliplu.1.0.2.dylib
+Uninstall broken packages by typing:
+    make -k antic-clean e_antic-clean lcalc-clean ratpoints-clean normaliz-clean r-clean thebe-clean asttokens-clean backports_zoneinfo-clean executing-clean fastjsonschema-clean pure_eval-clean pytz_deprecation_shim-clean stack_data-clean tinycss2-clean tzdata-clean suitesparse-clean
+
+real	0m6.671s
+user	0m33.285s
+sys	0m10.636s
+```
 
 The core functionality is implemented by the new command `sage-spkg-installcheck` (implemented in sage_bootstrap):

Answer 23 · 2022-07-21T02:02:37.000Z

comment:17

Ready for testing on macOS.

Answer 24 · 2022-07-21T02:19:15.000Z

Changed commit from 08388f7 to f2dba63

Answer 25 · 2022-07-21T02:19:15.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`f2dba63`	`build/sage_bootstrap/uninstall.py: Do not refuse to uninstall packages that do not exist in the source tree`

Answer 26 · 2022-07-21T03:03:32.000Z

Changed commit from f2dba63 to b439429

Answer 27 · 2022-07-21T03:03:32.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`a2437fc`	`build/sage_bootstrap/uninstall.py: Do not override the stamp file location from environment variable`
`b439429`	`build/make/Makefile.in (list-broken-packages): Show uninstallation commands that actually work`

Answer 28 · 2022-07-21T03:06:33.000Z

Description changed:

--- 
+++ 
@@ -17,7 +17,22 @@
 Checking shared library file '/Users/mkoeppe/s/sage/sage-rebasing/local/var/tmp/sage/build/suitesparse-5.10.1/src/lib/libsliplu.1.0.2.dylib'
 Error during installcheck of 'suitesparse': /Users/mkoeppe/s/sage/sage-rebasing/local/var/tmp/sage/build/suitesparse-5.10.1/src/lib/libsliplu.1.0.2.dylib
 Uninstall broken packages by typing:
-    make -k antic-clean e_antic-clean lcalc-clean ratpoints-clean normaliz-clean r-clean thebe-clean asttokens-clean backports_zoneinfo-clean executing-clean fastjsonschema-clean pure_eval-clean pytz_deprecation_shim-clean stack_data-clean tinycss2-clean tzdata-clean suitesparse-clean
+
+    ./sage --buildsh -c "sage-spkg-uninstall antic ";
+    ./sage --buildsh -c "sage-spkg-uninstall e_antic ";
+    ./sage --buildsh -c "sage-spkg-uninstall lcalc ";
+    ./sage --buildsh -c "sage-spkg-uninstall ratpoints ";
+    ./sage --buildsh -c "sage-spkg-uninstall normaliz ";
+    ./sage --buildsh -c "sage-spkg-uninstall r ";
+    ./sage --buildsh -c "sage-spkg-uninstall asttokens \$SAGE_VENV";
+    ./sage --buildsh -c "sage-spkg-uninstall backports_zoneinfo \$SAGE_VENV";
+    ./sage --buildsh -c "sage-spkg-uninstall executing \$SAGE_VENV";
+    ./sage --buildsh -c "sage-spkg-uninstall fastjsonschema \$SAGE_VENV";
+    ./sage --buildsh -c "sage-spkg-uninstall pytz_deprecation_shim \$SAGE_VENV";
+    ./sage --buildsh -c "sage-spkg-uninstall stack_data \$SAGE_VENV";
+    ./sage --buildsh -c "sage-spkg-uninstall tinycss2 \$SAGE_VENV";
+    ./sage --buildsh -c "sage-spkg-uninstall tzdata \$SAGE_VENV";
+    ./sage --buildsh -c "sage-spkg-uninstall suitesparse ";
 
 real	0m6.671s
 user	0m33.285s

Answer 29 · 2022-07-21T03:33:19.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`cd7312c`	`build/make/Makefile.in: New implicit targets %-SAGE_LOCAL-uninstall, %-SAGE_VENV-uninstall`
`5c104aa`	`build/make/Makefile.in (list-broken-packages): Use targets %-SAGE_LOCAL-uninstall, %-SAGE_VENV-uninstall when possible`

Answer 30 · 2022-07-21T03:33:19.000Z

Changed commit from b439429 to 5c104aa

Answer 31 · 2022-07-21T03:33:50.000Z

Description changed:

--- 
+++ 
@@ -18,21 +18,20 @@
 Error during installcheck of 'suitesparse': /Users/mkoeppe/s/sage/sage-rebasing/local/var/tmp/sage/build/suitesparse-5.10.1/src/lib/libsliplu.1.0.2.dylib
 Uninstall broken packages by typing:
 
-    ./sage --buildsh -c "sage-spkg-uninstall antic ";
-    ./sage --buildsh -c "sage-spkg-uninstall e_antic ";
-    ./sage --buildsh -c "sage-spkg-uninstall lcalc ";
-    ./sage --buildsh -c "sage-spkg-uninstall ratpoints ";
-    ./sage --buildsh -c "sage-spkg-uninstall normaliz ";
-    ./sage --buildsh -c "sage-spkg-uninstall r ";
-    ./sage --buildsh -c "sage-spkg-uninstall asttokens \$SAGE_VENV";
-    ./sage --buildsh -c "sage-spkg-uninstall backports_zoneinfo \$SAGE_VENV";
-    ./sage --buildsh -c "sage-spkg-uninstall executing \$SAGE_VENV";
-    ./sage --buildsh -c "sage-spkg-uninstall fastjsonschema \$SAGE_VENV";
-    ./sage --buildsh -c "sage-spkg-uninstall pytz_deprecation_shim \$SAGE_VENV";
-    ./sage --buildsh -c "sage-spkg-uninstall stack_data \$SAGE_VENV";
-    ./sage --buildsh -c "sage-spkg-uninstall tinycss2 \$SAGE_VENV";
-    ./sage --buildsh -c "sage-spkg-uninstall tzdata \$SAGE_VENV";
-    ./sage --buildsh -c "sage-spkg-uninstall suitesparse ";
+
+    make antic-SAGE_LOCAL-uninstall;
+    make e_antic-SAGE_LOCAL-uninstall;
+    make lcalc-SAGE_LOCAL-uninstall;
+    make ratpoints-SAGE_LOCAL-uninstall;
+    make normaliz-SAGE_LOCAL-uninstall;
+    make r-SAGE_LOCAL-uninstall;
+    make backports_zoneinfo-SAGE_VENV-uninstall;
+    make fastjsonschema-SAGE_VENV-uninstall;
+    make pytz_deprecation_shim-SAGE_VENV-uninstall;
+    make stack_data-SAGE_VENV-uninstall;
+    make tinycss2-SAGE_VENV-uninstall;
+    make tzdata-SAGE_VENV-uninstall;
+    make suitesparse-SAGE_LOCAL-uninstall;
 
 real	0m6.671s
 user	0m33.285s

Answer 32 · 2022-07-21T03:45:21.000Z

Changed commit from 5c104aa to 1d95bbb

Answer 33 · 2022-07-21T03:45:21.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`1d95bbb`	`build/make/Makefile.in (list-broken-packages): Fix up`

Answer 34 · 2022-07-21T03:45:47.000Z

Description changed:

--- 
+++ 
@@ -33,9 +33,9 @@
     make tzdata-SAGE_VENV-uninstall;
     make suitesparse-SAGE_LOCAL-uninstall;
 
-real	0m6.671s
-user	0m33.285s
-sys	0m10.636s
+real	2m25.787s
+user	4m54.270s
+sys	4m0.796s
 ```
 
 The core functionality is implemented by the new command `sage-spkg-installcheck` (implemented in sage_bootstrap):

Answer 35 · 2022-07-21T04:13:50.000Z

Description changed:

--- 
+++ 
@@ -1,6 +1,8 @@
-We can inspect the installation records in `$SAGE_LOCAL/var/lib/sage/installed/` by running `ldd`/`otool` on any shared libraries and uninstall them if they refer to nonexisting libraries.
+When system packages are upgraded, shared libraries that a Sage installation links to can disappear, rendering the Sage installation broken.
 
-The new command `make -j list-broken-packages` does this:
+To avoid a full rebuild (`make distclean && make build`), we can try to find the specific broken packages.
+
+The new command `make -j list-broken-packages` assists the user with this:
 
 ```
 $ make -j list-broken-packages
@@ -54,6 +56,10 @@
 Error during installcheck of 'e_antic': Could not find all dependencies.
 ```
 
+It inspects the installation records in `$SAGE_LOCAL/var/lib/sage/installed/` and all shared libraries. 
+
+
+
 On macOS, instead of using `otool` directly, we use `delocate-listdeps` from https://pypi.org/project/delocate/
 
 On Linux, instead of using `ldd` directly, we use `auditwheel show` from https://pypi.org/project/auditwheel/

Answer 36 · 2022-07-21T04:33:20.000Z

Description changed:

--- 
+++ 
@@ -62,7 +62,7 @@
 
 On macOS, instead of using `otool` directly, we use `delocate-listdeps` from https://pypi.org/project/delocate/
 
-On Linux, instead of using `ldd` directly, we use `auditwheel show` from https://pypi.org/project/auditwheel/
+On Linux, instead of using `ldd` directly, we can use `auditwheel show` from https://pypi.org/project/auditwheel/ -- follow-up ticket.
 
 (Both packages are also used by cibuildwheel)

Answer 37 · 2022-07-21T06:05:25.000Z

Changed commit from 1d95bbb to 37eaa92

Answer 38 · 2022-07-21T06:05:25.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`37eaa92`	`trac 29097: change "make SPKG-clean" to "make SPKG-uninstall".`

Answer 39 · 2022-07-21T06:06:17.000Z

comment:29

Cherry-picked / updated the commit from #29097.

Answer 40 · 2022-07-21T06:06:17.000Z

Changed author from Matthias Koeppe to Matthias Koeppe, John Palmieri

Answer 41 · 2022-07-21T06:32:32.000Z

Dependencies: #29097

Answer 42 · 2022-07-21T06:32:32.000Z

Changed author from Matthias Koeppe, John Palmieri to Matthias Koeppe

Answer 43 · 2022-07-21T06:39:25.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

`1458ee4`	`build/pkgs/auditwheel_or_delocate: New`
`8d1c6a4`	`build/sage_bootstrap/installcheck.py: Implement for macOS`
`320a290`	`build/sage_bootstrap/installcheck.py: Do not warn if there are no files to check`
`f19ff62`	`build/sage_bootstrap/installcheck.py: Skip pure Python wheels`
`53a98fd`	`build/make/Makefile.in (list-broken-packages): New`
`52bd73c`	`build/sage_bootstrap/installcheck.py: Only warn when delocate is not available`
`3bca243`	`build/make/Makefile.in (list-broken-packages): Parallelize it`
`e00a064`	`build/make/Makefile.in (list-broken-packages): Show uninstallation commands that actually work`
`980bbdb`	`build/make/Makefile.in (list-broken-packages): Use targets %-SAGE_LOCAL-uninstall, %-SAGE_VENV-uninstall when possible`
`1c24771`	`build/make/Makefile.in (list-broken-packages): Fix up`

Answer 44 · 2022-07-21T06:39:25.000Z

Changed commit from 37eaa92 to 1c24771

Answer 45 · 2022-07-21T08:09:05.000Z

Description changed:

--- 
+++ 
@@ -62,7 +62,7 @@
 
 On macOS, instead of using `otool` directly, we use `delocate-listdeps` from https://pypi.org/project/delocate/
 
-On Linux, instead of using `ldd` directly, we can use `auditwheel show` from https://pypi.org/project/auditwheel/ -- follow-up ticket.
+On Linux, instead of using `ldd` directly, we can use `auditwheel show` from https://pypi.org/project/auditwheel/ (https://github.com/pypa/auditwheel/blob/main/src/auditwheel/main_show.py) -- follow-up ticket.
 
 (Both packages are also used by cibuildwheel)

Answer 46 · 2022-07-21T16:30:31.000Z

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-When system packages are upgraded, shared libraries that a Sage installation links to can disappear, rendering the Sage installation broken.
+When system packages are upgraded, shared libraries that a Sage installation links to can disappear, rendering the Sage installation broken. This is a common problem, reported for example in https://groups.google.com/g/sage-devel/c/Iz8ZsmQM3Pg/m/CPl9CbHmBgAJ
 
 To avoid a full rebuild (`make distclean && make build`), we can try to find the specific broken packages.

Answer 47 · 2022-07-21T16:47:01.000Z

comment:34

This also finds a strange bug in our suitesparse installation on macOS:
The installation record (local/var/lib/sage/installed/suitesparse-5.10.1) contains paths in the temporary build directory (var/tmp/sage/build/suitesparse-5.10.1).
I've opened #34206 for this

Answer 48 · 2022-07-21T18:02:13.000Z

Description changed:

--- 
+++ 
@@ -66,4 +66,4 @@
 
 (Both packages are also used by cibuildwheel)
 
-
+See also: #34205 Optionally run auditwheel/delocate on wheels before installing them

Answer 49 · 2022-07-21T23:01:18.000Z

Changed commit from 1c24771 to 9d449bd

Answer 50 · 2022-07-21T23:01:18.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

`24fb9ff`	`build/sage_bootstrap/installcheck.py: Implement for macOS`
`3a4d2ed`	`build/sage_bootstrap/installcheck.py: Do not warn if there are no files to check`
`e36a56b`	`build/sage_bootstrap/installcheck.py: Skip pure Python wheels`
`bebc782`	`build/make/Makefile.in (list-broken-packages): New`
`0924cf1`	`build/sage_bootstrap/installcheck.py: Only warn when delocate is not available`
`d5473ec`	`build/make/Makefile.in (list-broken-packages): Parallelize it`
`75b1d96`	`build/make/Makefile.in (list-broken-packages): Show uninstallation commands that actually work`
`9418c08`	`build/make/Makefile.in (list-broken-packages): Use targets %-SAGE_LOCAL-uninstall, %-SAGE_VENV-uninstall when possible`
`51d2d56`	`build/make/Makefile.in (list-broken-packages): Fix up`
`9d449bd`	`build/sage_bootstrap/installclean.py: Clarify that the sage_local argument is SAGE_LOCAL or SAGE_VENV`

Answer 51 · 2022-07-21T23:10:11.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

`00ec17b`	`build/sage_bootstrap/installcheck.py: Implement for macOS`
`b2033b9`	`build/sage_bootstrap/installcheck.py: Do not warn if there are no files to check`
`eb47780`	`build/sage_bootstrap/installcheck.py: Skip pure Python wheels`
`30d9b5f`	`build/make/Makefile.in (list-broken-packages): New`
`e6679a7`	`build/sage_bootstrap/installcheck.py: Only warn when delocate is not available`
`3d94488`	`build/make/Makefile.in (list-broken-packages): Parallelize it`
`6881409`	`build/make/Makefile.in (list-broken-packages): Show uninstallation commands that actually work`
`b5a7d05`	`build/make/Makefile.in (list-broken-packages): Use targets %-SAGE_LOCAL-uninstall, %-SAGE_VENV-uninstall when possible`
`6a73497`	`build/make/Makefile.in (list-broken-packages): Fix up`
`400eaee`	`build/sage_bootstrap/installclean.py: Clarify that the sage_local argument is SAGE_LOCAL or SAGE_VENV`

Answer 52 · 2022-07-21T23:10:11.000Z

Changed commit from 9d449bd to 400eaee

Answer 53 · 2022-07-21T23:12:05.000Z

Description changed:

--- 
+++ 
@@ -56,7 +56,7 @@
 Error during installcheck of 'e_antic': Could not find all dependencies.
 ```
 
-It inspects the installation records in `$SAGE_LOCAL/var/lib/sage/installed/` and all shared libraries. 
+It inspects the installation records in `$SAGE_LOCAL/var/lib/sage/installed/` and all shared libraries and platform wheels listed there.

Answer 54 · 2022-07-23T19:33:13.000Z

Description changed:

--- 
+++ 
@@ -62,7 +62,7 @@
 
 On macOS, instead of using `otool` directly, we use `delocate-listdeps` from https://pypi.org/project/delocate/
 
-On Linux, instead of using `ldd` directly, we can use `auditwheel show` from https://pypi.org/project/auditwheel/ (https://github.com/pypa/auditwheel/blob/main/src/auditwheel/main_show.py) -- follow-up ticket.
+On Linux, instead of using `ldd` directly, we can use `auditwheel show` from https://pypi.org/project/auditwheel/ (https://github.com/pypa/auditwheel/blob/main/src/auditwheel/main_show.py) or `auditwheel.lddtree.lddtree` (https://github.com/pypa/auditwheel/blob/main/src/auditwheel/lddtree.py) -- follow-up ticket.
 
 (Both packages are also used by cibuildwheel)

Answer 55 · 2022-07-23T21:54:01.000Z

Changed commit from 400eaee to ef9226b

Answer 56 · 2022-07-23T21:54:01.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`ef9226b`	`build/sage_bootstrap/installcheck.py: Implement for Linux`

Answer 57 · 2022-07-23T21:54:55.000Z

Description changed:

--- 
+++ 
@@ -62,7 +62,7 @@
 
 On macOS, instead of using `otool` directly, we use `delocate-listdeps` from https://pypi.org/project/delocate/
 
-On Linux, instead of using `ldd` directly, we can use `auditwheel show` from https://pypi.org/project/auditwheel/ (https://github.com/pypa/auditwheel/blob/main/src/auditwheel/main_show.py) or `auditwheel.lddtree.lddtree` (https://github.com/pypa/auditwheel/blob/main/src/auditwheel/lddtree.py) -- follow-up ticket.
+On Linux, instead of using `ldd` directly, we use `auditwheel.lddtree.lddtree` (https://github.com/pypa/auditwheel/blob/main/src/auditwheel/lddtree.py)
 
 (Both packages are also used by cibuildwheel)

Answer 58 · 2022-08-22T23:04:52.000Z

comment:42

Hi Nils, I developed this ticket as a solution for last time that you complained about people breaking your build. Care to review?

Answer 59 · 2022-09-12T22:54:32.000Z

comment:44

Can you explain what "package" means in list-broken-packages? I ran the command and saw

usage: sage-spkg-installcheck [-h] [-v] spkg [sage_local]
sage-spkg-installcheck: error: argument spkg: 'stack_data' is not a known spkg

followed by the suggestion to run make stack_data-SAGE_VENV-uninstall. Is stack_data actually broken, or is the "usage" message indication of something else going wrong? And what is stack_data, since it's not an spkg?

Answer 60 · 2022-09-12T22:56:34.000Z

comment:45

Replying to John Palmieri:

Can you explain what "package" means in list-broken-packages?

An SPKG

Answer 61 · 2022-09-12T23:07:00.000Z

comment:46

Replying to John Palmieri:

Is stack_data actually broken, or is the "usage" message indication of something else going wrong? And what is stack_data, since it's not an spkg?

It's broken as follows: You have an installation of it (from some other branch), but it's not (per what is currently in build/pkgs) an SPKG.

Answer 62 · 2022-09-12T23:08:41.000Z

comment:47

In 9.7.rc1, stack_data of course is a package, so after merging 9.7.rc1, this will no longer be reported as "broken".

Answer 63 · 2022-09-12T23:08:56.000Z

Changed commit from ef9226b to 0bbba44

Answer 64 · 2022-09-12T23:08:56.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`0bbba44`	`Merge tag '9.7.rc1' into t/34203/find_and_uninstall_broken_installed_spkgs_and_wheels`

Answer 65 · 2022-09-12T23:09:21.000Z

comment:49

And here's that merge commit.

Answer 66 · 2022-09-13T00:27:51.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`9a03d7f`	`build/sage_bootstrap/installcheck.py: Make error message more specific`

Answer 67 · 2022-09-13T00:27:51.000Z

Changed commit from 0bbba44 to 9a03d7f

Answer 68 · 2022-09-14T21:33:09.000Z

comment:51

Should this have a nonzero return code if some packages are broken?

Answer 69 · 2022-09-14T21:36:57.000Z

comment:52

From its name list-broken-packages, I tend to say that it should not indicate a failure like that

Answer 70 · 2022-09-14T21:39:11.000Z

comment:53

I think I agree but I wanted to ask.

Answer 71 · 2022-09-14T21:40:11.000Z

comment:54

It is possible that some could want to do make -j list-broken-packages && do something else, but I don't see a concrete example.

Answer 72 · 2022-09-14T21:53:22.000Z

comment:55

Yes, I thought about this, but I'd say that this whole thing is a strictly manual operation, which doesn't lend itself to scripting.

Answer 73 · 2022-09-14T21:56:01.000Z

comment:56

For the new package auditwheel_or_delocate, it seems that potentially both are used on Linux, only delocate on OS X. If that's right, then the SPKG.rst should be modified.

Installing auditwheel via pip also installs pyelftools, and patchelf is listed as a dependency on the PyPI page. Do we need to create a new package pyelftools, too? Is patchelf really required, or is it only necessary if one is trying to fix files, not just detect whether they're broken?

Answer 74 · 2022-09-14T22:00:21.000Z

comment:57

The comment "and delete the rest" does not seem to be accurate (and I think it's fine not to delete the old stamp files here).

Answer 75 · 2022-09-14T22:15:37.000Z

Changed commit from 9a03d7f to 6ceb44e

Answer 76 · 2022-09-14T22:15:37.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`6ceb44e`	`build/pkgs/auditwheel_or_delocate/SPKG.rst: Explain that we use delocate also on Linux`

Answer 77 · 2022-09-14T22:21:49.000Z

comment:59

Replying to John Palmieri:

Installing auditwheel via pip also installs pyelftools, and patchelf is listed as a dependency on the PyPI page. Do we need to create a new package pyelftools, too?

Not needed - that's what makes optional "pip" packages so convenient.

Is patchelf really required, or is it only necessary if one is trying to fix files, not just detect whether they're broken?

Right, it's only needed for repairing wheels. But it's best not try to circumvent the declared dependencies.

The scipy upgrade (#34081) will bring a patchelf, by the way.

Answer 78 · 2022-09-14T22:24:13.000Z

Changed commit from 6ceb44e to 3c70a38

Answer 79 · 2022-09-14T22:24:13.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`3c70a38`	`build/sage_bootstrap/installcheck.py: Update comment on stamp files`

Answer 80 · 2022-09-14T22:30:50.000Z

comment:61

Great, I think this is ready to go now.

Answer 81 · 2022-09-14T22:30:50.000Z

Reviewer: John Palmieri

Answer 82 · 2022-09-14T22:36:45.000Z

comment:62

Thank you!

Answer 83 · 2022-09-20T20:22:59.000Z

Changed branch from u/mkoeppe/find_and_uninstall_broken_installed_spkgs_and_wheels to 3c70a38