New option "./configure --enable-wheels"
mkoeppe opened this issue · 104 comments
mkoeppe commented
When ./configure --disable-editable is in use (the default before #32406), the Sage library is installed using its custom incremental implementation of setup.py install -- which installs its files in site-packages and then removes the files that it does not know about (the "cleaner").
Problem 1: The cleaner is incompatible with namespace packages and trying to fix it would lead to complicated and awkward code (#32927). (Besides, setup.py install is deprecated since https://setuptools.pypa.io/en/latest/history.html#v58-3-0)
Problem 2: Because of the direct installation, we do not have wheels for sagemath-standard available (even if make wheels is used explicitly). Such wheels can be useful for making separate venvs.
In this ticket:
- We create a new option
./configure --enable-wheels(which can be used both with./configure --enable-editableand./configure --disable-editable). - If enabled, installation of sagelib will go through wheel building and installation - like we do with all other Python packages. This enables modularized builds (#34587).
- If disabled, there is no change.
A trivial ./sage -b is much slower with ./configure --disable-editable --enable-wheels compared to ./configure --disable-editable (from 28s as per timings in #32406 comment:21 to 70s).
This ticket brings a number of speed improvements to mitigate this.
CC: @jhpalmieri
Component: build
Author: Matthias Koeppe
Branch: 25898ba
Reviewer: Kwankyu Lee
Issue created by migration from https://trac.sagemath.org/ticket/32874
mkoeppe commented
mkoeppe commented
Description changed:
---
+++
@@ -1 +1,8 @@
+When `./configure --enable-editable` has not been used, the Sage library is currently installed using its custom incremental implementation of `setup.py install` -- which installs its files in `site-packages` and then removes the files that it does not know about (the "cleaner").
+This is incompatible with namespace packages. (Besides, `setup.py install` is deprecated since https://setuptools.pypa.io/en/latest/history.html#v58-3-0)
+
+In this ticket, we change the installation procedure to go through wheel building and installation. This makes a trivial `./sage -b` much slower (from 10s to 40s).
+
+Developers can switch to using `./configure --enable-editable`, for which a trivial `./sage -b` takes 17s. (#32406 proposes to make this the default.)
+mkoeppe commented
New commits:
b9b1dba | Enable editable mode also for other sage packages |
a257621 | Add error handling |
89bc3ee | Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace_ext |
7bd6ce4 | Partly revert "Enable editable mode also for other sage packages" |
3ef9c1d | Merge #32713 |
50dd26c | build/pkgs/sagelib/spkg-install: Use helper function sdh_pip_editable_install |
b5c40a8 | build/pkgs/sagelib/spkg-install: In non-editable mode, Use sdh_pip_install instead of 'setup.py install' |
mkoeppe commented
Author: Matthias Koeppe
mkoeppe commented
Description changed:
---
+++
@@ -6,3 +6,5 @@
Developers can switch to using `./configure --enable-editable`, for which a trivial `./sage -b` takes 17s. (#32406 proposes to make this the default.)
+
+An alternative solution: #32927.sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
8619b29 | configure.ac: Make --enable-editable the default |
6513b6b | README.md: Explain configure --disable-editable |
ff8710e | docker/Dockerfile: Use configure --disable-editable |
7221a76 | src/setup.py: Do not run find_namespace_packages for 'setup.py dist_info' |
a911e0f | src/MANIFEST.in: prune sage_docbuild, doc |
edfa3b4 | build/pkgs/sagelib/spkg-install: Use helper function sdh_pip_editable_install |
f7ea602 | build/pkgs/sagelib/spkg-install: In non-editable mode, Use sdh_pip_install instead of 'setup.py install' |
sagetrac-git commented
mkoeppe commented
Description changed:
---
+++
@@ -1,10 +1,9 @@
-When `./configure --enable-editable` has not been used, the Sage library is currently installed using its custom incremental implementation of `setup.py install` -- which installs its files in `site-packages` and then removes the files that it does not know about (the "cleaner").
+When `./configure --disable-editable` is in use (the default before #32406), the Sage library is installed using its custom incremental implementation of `setup.py install` -- which installs its files in `site-packages` and then removes the files that it does not know about (the "cleaner").
-This is incompatible with namespace packages. (Besides, `setup.py install` is deprecated since https://setuptools.pypa.io/en/latest/history.html#v58-3-0)
+The cleaner is incompatible with namespace packages and trying to fix it would lead to complicated and awkward code (#32927). (Besides, `setup.py install` is deprecated since https://setuptools.pypa.io/en/latest/history.html#v58-3-0)
-In this ticket, we change the installation procedure to go through wheel building and installation. This makes a trivial `./sage -b` much slower (from 10s to 40s).
+In this ticket, we change the installation procedure to go through wheel building and installation - like we do with all other Python packages.
-Developers can switch to using `./configure --enable-editable`, for which a trivial `./sage -b` takes 17s. (#32406 proposes to make this the default.)
+However, this makes a trivial `./sage -b` much slower (from 10s to 40s). This should be acceptable because after #32406, the primary use case of `./configure --disable-editable` is for fixed installations in an installation prefix, not for development.
-An alternative solution: #32927.mkoeppe commented
comment:10
[sagelib-9.7.beta5] sage-flock: error: argument LOCK: can't create '/doesnotexist/var/lock': [Errno 30] Read-only file system: '/doesnotexist'
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
df1e7eb | build/bin/sage-pip-install: Put lock file in SAGE_VENV, not SAGE_LOCAL |
f358962 | build/pkgs/sagelib/spkg-install: No need to set env var PYTHON |
c761788 | build/bin/sage-pip-uninstall: Put lock file in SAGE_VENV, not SAGE_LOCAL |
99d4129 | build/pkgs/sagelib/spkg-install [SAGE_EDITABLE=no]: Use --no-build-isolation |
sagetrac-git commented
mkoeppe commented
Description changed:
---
+++
@@ -4,6 +4,6 @@
In this ticket, we change the installation procedure to go through wheel building and installation - like we do with all other Python packages.
-However, this makes a trivial `./sage -b` much slower (from 10s to 40s). This should be acceptable because after #32406, the primary use case of `./configure --disable-editable` is for fixed installations in an installation prefix, not for development.
+However, this makes a trivial `./sage -b` much slower (from 28s as per timings in [#32406 comment:21](https://github.com/sagemath/sage/issues/32406#comment:21) to 70s). This should be acceptable because after #32406, the primary use case of `./configure --disable-editable` is for fixed installations in an installation prefix, not for development.
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
07f5cde | build/pkgs/sagelib/spkg-install: Use helper function sdh_pip_editable_install |
3cdf86e | build/pkgs/sagelib/spkg-install: In non-editable mode, Use sdh_pip_install instead of 'setup.py install' |
459ee2f | build/bin/sage-pip-install: Put lock file in SAGE_VENV, not SAGE_LOCAL |
3de2608 | build/pkgs/sagelib/spkg-install: No need to set env var PYTHON |
e6eae68 | build/bin/sage-pip-uninstall: Put lock file in SAGE_VENV, not SAGE_LOCAL |
e829b08 | build/pkgs/sagelib/spkg-install [SAGE_EDITABLE=no]: Use --no-build-isolation |
sagetrac-git commented
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
31bda8e | src/sage_setup/command/sage_install.py: Do not attempt to clean the install directory |
sagetrac-git commented
kwankyu commented
comment:18
So with this patch, in --disable-editable mode, why sage -b gets so slow? Doesn't it do incremental build?
With this patch, in --enable-editable mode, sage -b does not get slower?
Could we be sure that developers do not use --disable-editable for development? If there are, you are going to persuade them not to use it for development?
mkoeppe commented
comment:19
Replying to Kwankyu Lee:
So with this patch, in
--disable-editablemode, whysage -bgets so slow? Doesn't it do incremental build?
The build is still incremental, but it's packing everything in a wheel (90 MB) and unpacking the wheel when installing it, which I think accounts for the slowdown. But I have not really attempted to run this with a profiler or anything.
With this patch, in
--enable-editablemode,sage -bdoes not get slower?
No, there's no change to that mode.
Could we be sure that developers do not use
--disable-editablefor development?
Hard to be sure.
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
e97c237 | Merge tag '9.8.beta1' into t/32874/remove_use_of__setup_py_install__for_sagelib |
sagetrac-git commented
mkoeppe commented
comment:21
Some more timings to see where the total of 60 seconds is spent in a trivial ./sage -b:
-
To uninstall the previous installation of sagelib takes about 3 seconds (timed with
./sage -sh -c 'time pip uninstall --yes sagemath-standard') -
To install the wheel takes about 10 seconds (timed with
./sage -sh -c 'time pip install venv/var/lib/sage/wheels/sagemath_standard-9.8b1-cp310-cp310-macosx_12_0_x86_64.whl')
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
a3d8c51 | sage_setup.find.find_python_sources: Do not recurse into subdirectories that are not packages/namespace packages |
40f8826 | sage_setup.find: Refactor through new class sage.misc.package_dir.SourceDistributionFilter |
bf6e54c | src/sage_setup/find.py, src/sage_setup/clean.py, src/sage/misc/package_dir.py: Update copyright according to "git blame -w --date=format:%Y FILE | sort -k2" |
sagetrac-git commented
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
7c98f0e | pkgs/sagemath-standard/setup.py: Do not run source file discovery on 'setup.py distinfo' |
sagetrac-git commented
mkoeppe commented
comment:24
The last commit shaves off about 5 seconds.
mkoeppe commented
comment:25
The wheel-building part itself is about 31 seconds
(sage-buildsh) mkoeppe@egret:~/s/sage/sage-rebasing/worktree-rebase/pkgs/sagemath-standard$ time python3 setup.py bdist_wheel
23.91s user 6.71s system 99% cpu 30.793 total
mkoeppe commented
comment:26
Here's the result of pasting the top of pkgs/sagemath-standard/setup.py in sage -ipython and then running %prun setup(packages = python_packages, cmdclass = cmdclass, ext_modules = cython_modules)
21884054 function calls (20880049 primitive calls) in 34.998 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
4737 10.807 0.002 10.807 0.002 {method 'compress' of 'zlib.Compress' objects}
567281 2.715 0.000 2.715 0.000 {built-in method posix.stat}
1485393 2.480 0.000 3.887 0.000 posixpath.py:71(join)
64029 1.027 0.000 1.027 0.000 {method 'read' of '_io.BufferedReader' objects}
56924 0.893 0.000 0.893 0.000 {built-in method posix.getcwd}
19375 0.843 0.000 0.858 0.000 {built-in method io.open}
4736 0.837 0.000 0.837 0.000 {built-in method _hashlib.openssl_sha256}
25992 0.581 0.000 0.581 0.000 {method 'write' of '_io.BufferedWriter' objects}
88515 0.506 0.000 0.742 0.000 posixpath.py:337(normpath)
208247 0.501 0.000 4.150 0.000 Utils.py:149(contains_init)
4737 0.471 0.000 0.471 0.000 {method 'flush' of 'zlib.Compress' objects}
961672/52290 0.470 0.000 8.841 0.000 Utils.py:38(wrapper)
1948728 0.441 0.000 0.441 0.000 {method 'startswith' of 'str' objects}
1643384 0.438 0.000 0.665 0.000 posixpath.py:41(_get_sep)
4762 0.373 0.000 0.373 0.000 {built-in method posix.unlink}
143 0.348 0.002 0.348 0.002 {built-in method posix.waitpid}
984573 0.345 0.000 0.345 0.000 {method 'get' of 'dict' objects}
497648 0.339 0.000 2.592 0.000 genericpath.py:16(exists)
4750 0.339 0.000 0.339 0.000 {method 'close' of '_io.BufferedWriter' objects}
1853 0.337 0.000 0.565 0.000 Dependencies.py:308(strip_string_literals)
141 0.337 0.002 0.337 0.002 {method 'poll' of 'select.poll' objects}
2083096 0.295 0.000 0.297 0.000 {built-in method builtins.isinstance}
1683819 0.293 0.000 0.293 0.000 {method 'endswith' of 'str' objects}
81972 0.271 0.000 0.271 0.000 {built-in method posix.lstat}
8433 0.246 0.000 7.077 0.001 Main.py:787(search_include_directories)
399961 0.241 0.000 0.241 0.000 {method 'match' of 're.Pattern' objects}
3118 0.219 0.000 0.236 0.000 {method 'read' of '_io.TextIOWrapper' objects}
1200 0.218 0.000 1.459 0.001 Dependencies.py:480(parse_dependencies)
1921596 0.197 0.000 0.197 0.000 {built-in method posix.fspath}
355 0.185 0.001 0.185 0.001 {built-in method posix.read}
253507 0.171 0.000 0.171 0.000 {method 'find' of 'str' objects}
219811 0.160 0.000 1.225 0.000 Utils.py:165(path_exists)
142 0.160 0.001 0.160 0.001 {built-in method _posixsubprocess.fork_exec}
37898 0.152 0.000 0.152 0.000 {method 'write' of '_io.BufferedRandom' objects}
10971/5491 0.150 0.000 0.649 0.000 posixpath.py:400(_joinrealpath)
26680 0.144 0.000 1.897 0.000 posixpath.py:465(relpath)
37406 0.143 0.000 4.785 0.000 Utils.py:138(check_package_dir)
1363000 0.143 0.000 0.143 0.000 {method 'append' of 'list' objects}
75046 0.133 0.000 1.938 0.000 posixpath.py:376(abspath)
4756 0.128 0.000 0.128 0.000 {built-in method posix.utime}
1 0.117 0.117 13.865 13.865 wheelfile.py:120(write_files)
53941 0.110 0.000 0.145 0.000 {built-in method builtins.next}
8373 0.102 0.000 2.423 0.000 file_util.py:70(copy_file)
183237 0.097 0.000 0.097 0.000 {method 'split' of 'str' objects}
183237 0.097 0.000 0.097 0.000 {method 'split' of 'str' objects}
11009 0.096 0.000 0.521 0.000 tarfile.py:1054(frombuf)
143104 0.092 0.000 0.146 0.000 tarfile.py:164(nts)
4885 0.089 0.000 0.089 0.000 {built-in method posix.chmod}
4455 0.086 0.000 0.086 0.000 {built-in method posix.scandir}
93 0.078 0.001 0.141 0.002 Code.py:273(load_utilities_from_file)
22240 0.076 0.000 0.076 0.000 {built-in method builtins.sum}
88064 0.075 0.000 0.173 0.000 tarfile.py:172(nti)
4383/4372 0.075 0.000 0.097 0.000 {built-in method builtins.__build_class__}
39067 0.075 0.000 0.348 0.000 tarfile.py:553(__read)
90395 0.071 0.000 0.137 0.000 posixpath.py:60(isabs)
26680 0.066 0.000 0.101 0.000 genericpath.py:69(commonprefix)
33569 0.065 0.000 0.065 0.000 {built-in method __new__ of type object at 0x103f489f0}
17411/3144 0.065 0.000 0.266 0.000 os.py:344(_walk)
14213 0.064 0.000 0.064 0.000 {method 'seek' of '_io.BufferedRandom' objects}
4747 0.062 0.000 1.738 0.000 file_util.py:14(_copy_file_contents)
22016 0.060 0.000 0.060 0.000 {built-in method _struct.unpack_from}
313225/313193 0.059 0.000 0.061 0.000 {built-in method builtins.getattr}
4737 0.057 0.000 0.697 0.000 zipfile.py:1142(close)
145319/145148 0.056 0.000 0.057 0.000 {method 'join' of 'str' objects}
24132 0.056 0.000 0.076 0.000 glob.py:128(_iterdir)
442 0.053 0.000 0.053 0.000 {built-in method marshal.loads}
25789 0.052 0.000 0.079 0.000 posixpath.py:100(split)
189632 0.052 0.000 0.052 0.000 {built-in method builtins.min}
744 0.051 0.000 0.058 0.000 intrinsic.py:52(__init__)
12933 0.050 0.000 0.747 0.000 {method '__exit__' of '_io._IOBase' objects}
21222 0.050 0.000 0.050 0.000 {method 'search' of 're.Pattern' objects}
9474 0.046 0.000 0.080 0.000 zipfile.py:409(FileHeader)
499 0.045 0.000 0.045 0.000 {built-in method posix.listdir}
14 0.045 0.003 0.229 0.016 egg_info.py:477(<listcomp>)
142 0.044 0.000 0.570 0.004 subprocess.py:756(__init__)
15541 0.041 0.000 0.065 0.000 posixpath.py:150(dirname)
7 0.041 0.006 0.086 0.012 egg_info.py:413(_remove_files)
4736 0.040 0.000 13.143 0.003 wheelfile.py:142(write)
139 0.040 0.000 0.041 0.000 warnings.py:458(__enter__)
315793/313610 0.039 0.000 0.040 0.000 {built-in method builtins.len}
36421/19887 0.039 0.000 7.654 0.000 Utils.py:48(wrapper)
142 0.038 0.000 0.503 0.004 subprocess.py:1685(_execute_child)
4737 0.037 0.000 12.771 0.003 wheelfile.py:152(writestr)
230/2 0.036 0.000 2.478 1.239 dir_util.py:107(copy_tree)
11009 0.035 0.000 1.085 0.000 tarfile.py:2329(next)
4737 0.034 0.000 11.852 0.003 zipfile.py:1776(writestr)
43 0.034 0.001 0.034 0.001 {built-in method _imp.create_dynamic}
87871 0.033 0.000 0.042 0.000 {built-in method builtins.max}
10500 0.033 0.000 7.281 0.001 Dependencies.py:572(find_pxd)
26262 0.033 0.000 0.058 0.000 posixpath.py:140(basename)
29932 0.032 0.000 0.217 0.000 genericpath.py:27(isfile)
2 0.032 0.016 1.543 0.771 filelist.py:302(findall)
106344 0.032 0.000 0.032 0.000 {method 'rfind' of 'str' objects}
39067 0.030 0.000 0.407 0.000 tarfile.py:519(read)
11008 0.030 0.000 0.165 0.000 tarfile.py:222(calc_chksums)
152480 0.030 0.000 0.030 0.000 {method 'decode' of 'bytes' objects}
24048 0.030 0.000 0.185 0.000 egg_info.py:509(_safe_path)
143104 0.029 0.000 0.029 0.000 {method 'find' of 'bytes' objects}
4886 0.028 0.000 0.028 0.000 {method 'close' of '_io.BufferedReader' objects}
4737 0.028 0.000 0.127 0.000 zipfile.py:1576(_open_to_write)
70 0.027 0.000 0.425 0.006 subprocess.py:1950(_communicate)
4737 0.027 0.000 0.027 0.000 {built-in method zlib.crc32}
4737 0.027 0.000 10.958 0.002 zipfile.py:1123(write)
19639 0.026 0.000 7.160 0.000 Main.py:241(find_pxd_file)
26680 0.026 0.000 0.026 0.000 posixpath.py:488(<listcomp>)
10431 0.026 0.000 0.077 0.000 <frozen importlib._bootstrap_external>:380(cache_from_source)
39067 0.024 0.000 0.372 0.000 tarfile.py:526(_read)
11007 0.024 0.000 0.061 0.000 tarfile.py:1151(_proc_builtin)
19674 0.024 0.000 7.131 0.000 Main.py:288(search_include_directories)
20696/7591 0.024 0.000 0.040 0.000 signature.py:5(type_dependencies)
4737 0.023 0.000 0.023 0.000 {built-in method zlib.compressobj}
442 0.023 0.000 0.023 0.000 {built-in method io.open_code}
1949 0.022 0.000 0.022 0.000 {built-in method posix.mkdir}
26680 0.022 0.000 0.022 0.000 posixpath.py:487(<listcomp>)
88965 0.022 0.000 0.022 0.000 {method 'partition' of 'str' objects}
11009/11008 0.021 0.000 0.655 0.000 tarfile.py:1117(fromtarfile)
17120 0.021 0.000 0.157 0.000 genericpath.py:39(isdir)
6047 0.020 0.000 0.022 0.000 Dependencies.py:258(merge)
12094/1068 0.020 0.000 9.012 0.008 Dependencies.py:697(transitive_merge_helper)
1 0.020 0.020 1.000 1.000 file_finder.py:6(scm_find_files)
1068 0.020 0.000 0.106 0.000 macosx_libfile.py:305(read_mach_header)
14228 0.020 0.000 0.026 0.000 genericpath.py:121(_splitext)
792 0.019 0.000 0.540 0.001 build_py.py:220(find_package_modules)
9384 0.019 0.000 0.362 0.000 tarfile.py:506(seek)
64 0.018 0.000 0.678 0.011 pkgconfig.py:106(_query)
239/2 0.018 0.000 0.434 0.217 shutil.py:626(_rmtree_safe_fd)
1214 0.018 0.000 0.370 0.000 Utils.py:239(open_source_file)
8688 0.018 0.000 0.032 0.000 os.py:674(__getitem__)
66 0.018 0.000 0.631 0.010 subprocess.py:337(call)
6623 0.017 0.000 0.044 0.000 _collections_abc.py:816(get)
2268 0.017 0.000 0.039 0.000 Dependencies.py:226(__init__)
8355 0.017 0.000 0.118 0.000 dep_util.py:11(newer)
4291 0.017 0.000 0.318 0.000 {built-in method builtins.sorted}
mkoeppe commented
comment:27
Most of the time is spent in
- Cython's dependency tracking
search_include_directories(https://github.com/cython/cython/blob/0.29.x/Cython/Compiler/Main.py),transitive_merge_helper(https://github.com/cython/cython/blob/0.29.x/Cython/Build/Dependencies.py) (about 16 seconds) - compressed wheel file writing (about 13 seconds)
Our setup.py also spends about 5 seconds in Python package/module discovery.
It would be worth checking if we can use an on-disk cache to speed up dependency tracking and package/module discovery.
mkoeppe commented
comment:28
Related: cython/cython#4949 (cython --depfile), mesonbuild/meson#9049 (Cython dep handling), https://github.com/scipy/scipy/blob/main/tools/cythonize.py, https://github.com/scipy/scipy/blob/main/scipy/linalg/meson.build
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
a3d5c3e | sage.misc.package_dir.cython_namespace_package_support: Use Cython.Utils.cached_function |
sagetrac-git commented
mkoeppe commented
comment:30
This one simple trick shaves off 13 seconds
mkoeppe commented
comment:31
21787023 function calls (20782202 primitive calls) in 30.557 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
4737 10.194 0.002 10.194 0.002 {method 'compress' of 'zlib.Compress' objects}
564497 2.160 0.000 2.160 0.000 {built-in method posix.stat}
1480789 2.063 0.000 3.314 0.000 posixpath.py:71(join)
55724 0.769 0.000 0.769 0.000 {built-in method posix.getcwd}
4736 0.762 0.000 0.762 0.000 {built-in method _hashlib.openssl_sha256}
19375 0.736 0.000 0.748 0.000 {built-in method io.open}
64029 0.599 0.000 0.599 0.000 {method 'read' of '_io.BufferedReader' objects}
25992 0.479 0.000 0.479 0.000 {method 'write' of '_io.BufferedWriter' objects}
1937517 0.447 0.000 0.447 0.000 {method 'startswith' of 'str' objects}
87416 0.443 0.000 0.651 0.000 posixpath.py:337(normpath)
4737 0.429 0.000 0.429 0.000 {method 'flush' of 'zlib.Compress' objects}
4762 0.414 0.000 0.414 0.000 {built-in method posix.unlink}
208247 0.408 0.000 3.368 0.000 Utils.py:149(contains_init)
963346/52290 0.398 0.000 6.816 0.000 Utils.py:38(wrapper)
1637573 0.371 0.000 0.558 0.000 posixpath.py:41(_get_sep)
143 0.341 0.002 0.341 0.002 {built-in method posix.waitpid}
140 0.330 0.002 0.330 0.002 {method 'poll' of 'select.poll' objects}
4750 0.292 0.000 0.292 0.000 {method 'close' of '_io.BufferedWriter' objects}
986247 0.283 0.000 0.283 0.000 {method 'get' of 'dict' objects}
495577 0.282 0.000 2.057 0.000 genericpath.py:16(exists)
1853 0.266 0.000 0.461 0.000 Dependencies.py:308(strip_string_literals)
81972 0.259 0.000 0.259 0.000 {built-in method posix.lstat}
2073736 0.245 0.000 0.248 0.000 {built-in method builtins.isinstance}
1676002 0.244 0.000 0.244 0.000 {method 'endswith' of 'str' objects}
8433 0.203 0.000 5.744 0.001 Main.py:787(search_include_directories)
390060 0.188 0.000 0.188 0.000 {method 'match' of 're.Pattern' objects}
1200 0.171 0.000 0.795 0.001 Dependencies.py:480(parse_dependencies)
1912273 0.169 0.000 0.169 0.000 {built-in method posix.fspath}
356 0.166 0.000 0.166 0.000 {built-in method posix.read}
142 0.157 0.001 0.157 0.001 {built-in method _posixsubprocess.fork_exec}
253507 0.151 0.000 0.151 0.000 {method 'find' of 'str' objects}
10971/5491 0.144 0.000 0.621 0.000 posixpath.py:400(_joinrealpath)
219873 0.131 0.000 0.930 0.000 Utils.py:165(path_exists)
26080 0.129 0.000 1.684 0.000 posixpath.py:465(relpath)
1349185 0.126 0.000 0.126 0.000 {method 'append' of 'list' objects}
37406 0.119 0.000 3.900 0.000 Utils.py:138(check_package_dir)
73846 0.118 0.000 1.705 0.000 posixpath.py:376(abspath)
1 0.099 0.099 12.917 12.917 wheelfile.py:120(write_files)
53113 0.096 0.000 0.128 0.000 {built-in method builtins.next}
4756 0.094 0.000 0.094 0.000 {built-in method posix.utime}
37898 0.092 0.000 0.092 0.000 {method 'write' of '_io.BufferedRandom' objects}
11009 0.089 0.000 0.472 0.000 tarfile.py:1054(frombuf)
8373 0.088 0.000 2.092 0.000 file_util.py:70(copy_file)
180938 0.084 0.000 0.084 0.000 {method 'split' of 'str' objects}
143104 0.082 0.000 0.130 0.000 tarfile.py:164(nts)
4885 0.078 0.000 0.078 0.000 {built-in method posix.chmod}
4329 0.075 0.000 0.075 0.000 {built-in method posix.scandir}
39067 0.074 0.000 0.352 0.000 tarfile.py:553(__read)
442 0.073 0.000 0.073 0.000 {built-in method marshal.loads}
88064 0.069 0.000 0.158 0.000 tarfile.py:172(nti)
4383/4372 0.067 0.000 0.082 0.000 {built-in method builtins.__build_class__}
22240 0.066 0.000 0.066 0.000 {built-in method builtins.sum}
89195 0.062 0.000 0.122 0.000 posixpath.py:60(isabs)
93 0.062 0.001 0.103 0.001 Code.py:273(load_utilities_from_file)
14213 0.060 0.000 0.060 0.000 {method 'seek' of '_io.BufferedRandom' objects}
26080 0.059 0.000 0.088 0.000 genericpath.py:69(commonprefix)
16444/3031 0.058 0.000 0.233 0.000 os.py:344(_walk)
33569 0.056 0.000 0.056 0.000 {built-in method __new__ of type object at 0x109d009f0}
790 0.056 0.000 0.056 0.000 functools.py:65(wraps)
22016 0.055 0.000 0.055 0.000 {built-in method _struct.unpack_from}
4747 0.054 0.000 1.499 0.000 file_util.py:14(_copy_file_contents)
313287/313255 0.051 0.000 0.053 0.000 {built-in method builtins.getattr}
4737 0.051 0.000 0.629 0.000 zipfile.py:1142(close)
144220/144049 0.047 0.000 0.048 0.000 {method 'join' of 'str' objects}
25782 0.045 0.000 0.068 0.000 posixpath.py:100(split)
12933 0.041 0.000 0.670 0.000 {method '__exit__' of '_io._IOBase' objects}
189025 0.041 0.000 0.041 0.000 {built-in method builtins.min}
9474 0.041 0.000 0.071 0.000 zipfile.py:409(FileHeader)
7 0.040 0.006 0.076 0.011 egg_info.py:413(_remove_files)
22618 0.040 0.000 0.057 0.000 glob.py:128(_iterdir)
142 0.039 0.000 0.469 0.003 subprocess.py:1685(_execute_child)
21183 0.038 0.000 0.038 0.000 {method 'search' of 're.Pattern' objects}
4736 0.036 0.000 12.267 0.003 wheelfile.py:142(write)
314554/312374 0.035 0.000 0.036 0.000 {built-in method builtins.len}
15541 0.035 0.000 0.056 0.000 posixpath.py:150(dirname)
43 0.035 0.001 0.035 0.001 {built-in method _imp.create_dynamic}
14 0.034 0.002 0.176 0.013 egg_info.py:477(<listcomp>)
4737 0.033 0.000 11.929 0.003 wheelfile.py:152(writestr)
230/2 0.032 0.000 2.142 1.071 dir_util.py:107(copy_tree)
11009 0.032 0.000 1.025 0.000 tarfile.py:2329(next)
142 0.031 0.000 0.526 0.004 subprocess.py:756(__init__)
4737 0.031 0.000 11.095 0.002 zipfile.py:1776(writestr)
36421/19887 0.030 0.000 6.168 0.000 Utils.py:48(wrapper)
39067 0.029 0.000 0.408 0.000 tarfile.py:519(read)
26262 0.029 0.000 0.051 0.000 posixpath.py:140(basename)
2 0.029 0.014 1.346 0.673 filelist.py:302(findall)
87271 0.029 0.000 0.036 0.000 {built-in method builtins.max}
29332 0.028 0.000 0.192 0.000 genericpath.py:27(isfile)
11008 0.028 0.000 0.149 0.000 tarfile.py:222(calc_chksums)
1955 0.028 0.000 0.028 0.000 {built-in method posix.mkdir}
152480 0.027 0.000 0.027 0.000 {method 'decode' of 'bytes' objects}
106337 0.027 0.000 0.027 0.000 {method 'rfind' of 'str' objects}
499 0.027 0.000 0.027 0.000 {built-in method posix.listdir}
4886 0.026 0.000 0.026 0.000 {method 'close' of '_io.BufferedReader' objects}
66 0.026 0.000 0.614 0.009 subprocess.py:337(call)
143104 0.025 0.000 0.025 0.000 {method 'find' of 'bytes' objects}
10500 0.025 0.000 5.883 0.001 Dependencies.py:572(find_pxd)
24048 0.025 0.000 0.152 0.000 egg_info.py:509(_safe_path)
139 0.024 0.000 0.025 0.000 warnings.py:458(__enter__)
4737 0.024 0.000 0.024 0.000 {built-in method zlib.crc32}
4737 0.024 0.000 0.114 0.000 zipfile.py:1576(_open_to_write)
26080 0.023 0.000 0.023 0.000 posixpath.py:488(<listcomp>)
4737 0.023 0.000 10.287 0.002 zipfile.py:1123(write)
8688 0.023 0.000 0.037 0.000 os.py:674(__getitem__)
39067 0.022 0.000 0.375 0.000 tarfile.py:526(_read)
10431 0.022 0.000 0.068 0.000 <frozen importlib._bootstrap_external>:380(cache_from_source)
11007 0.022 0.000 0.055 0.000 tarfile.py:1151(_proc_builtin)
4737 0.021 0.000 0.021 0.000 {built-in method zlib.compressobj}
3118 0.021 0.000 0.032 0.000 {method 'read' of '_io.TextIOWrapper' objects}
88965 0.021 0.000 0.021 0.000 {method 'partition' of 'str' objects}
19639 0.020 0.000 5.809 0.000 Main.py:241(find_pxd_file)
239/2 0.020 0.000 0.481 0.240 shutil.py:626(_rmtree_safe_fd)
11009/11008 0.020 0.000 0.597 0.000 tarfile.py:1117(fromtarfile)
26080 0.019 0.000 0.019 0.000 posixpath.py:487(<listcomp>)
17120 0.019 0.000 0.141 0.000 genericpath.py:39(isdir)
70 0.019 0.000 0.408 0.006 subprocess.py:1950(_communicate)
19674 0.019 0.000 5.786 0.000 Main.py:288(search_include_directories)
1 0.019 0.019 0.949 0.949 file_finder.py:6(scm_find_files)
1068 0.018 0.000 0.099 0.000 macosx_libfile.py:305(read_mach_header)
9384 0.018 0.000 0.365 0.000 tarfile.py:506(seek)
130 0.018 0.000 1.309 0.010 pkgconfig.py:88(_wrapper)
64 0.017 0.000 0.660 0.010 pkgconfig.py:106(_query)
792 0.017 0.000 0.456 0.001 build_py.py:220(find_package_modules)
14228 0.017 0.000 0.022 0.000 genericpath.py:121(_splitext)
442 0.017 0.000 0.017 0.000 {built-in method io.open_code}
20696/7591 0.017 0.000 0.029 0.000 signature.py:5(type_dependencies)
1 0.016 0.016 0.025 0.025 zipfile.py:1843(_write_end_record)
12094/1068 0.016 0.000 6.872 0.006 Dependencies.py:697(transitive_merge_helper)
6047 0.016 0.000 0.017 0.000 Dependencies.py:258(merge)
72 0.016 0.000 0.460 0.006 subprocess.py:1108(communicate)
8355 0.015 0.000 0.107 0.000 dep_util.py:11(newer)
4737 0.014 0.000 0.014 0.000 {built-in method time.gmtime}
11007 0.014 0.000 0.019 0.000 tarfile.py:1365(_apply_pax_info)
4695 0.014 0.000 0.056 0.000 find.py:393(add)
52176 0.014 0.000 0.014 0.000 {method 'encode' of 'str' objects}
39207 0.014 0.000 0.014 0.000 {method 'join' of 'bytes' objects}
5483 0.014 0.000 0.014 0.000 {built-in method posix.readlink}
4291 0.013 0.000 0.253 0.000 {built-in method builtins.sorted}
4737 0.013 0.000 0.016 0.000 zipfile.py:344(__init__)
23103 0.013 0.000 0.013 0.000 {method 'replace' of 'str' objects}
14217 0.013 0.000 0.013 0.000 {built-in method _struct.pack}
14228 0.012 0.000 0.037 0.000 posixpath.py:117(splitext)
81433 0.012 0.000 0.012 0.000 {method 'rstrip' of 'str' objects}
1411 0.012 0.000 5.921 0.004 Dependencies.py:596(cimported_files)
4737 0.012 0.000 0.051 0.000 wheelfile.py:35(get_zipinfo_datetime)
15780 0.012 0.000 0.141 0.000 filelist.py:269(<genexpr>)
13119 0.012 0.000 0.015 0.000 sre_parse.py:1066(expand_template)
4212/91 0.012 0.000 0.045 0.000 core.py:776(_parseNoCache)
142 0.012 0.000 0.031 0.000 subprocess.py:1225(_close_pipe_fds)
4975 0.012 0.000 0.012 0.000 {built-in method posix.fstat}
4737 0.011 0.000 0.139 0.000 wheelfile.py:93(open)
11008 0.011 0.000 0.011 0.000 tarfile.py:750(__init__)
100970 0.011 0.000 0.011 0.000 {method 'strip' of 'str' objects}
2268 0.011 0.000 0.026 0.000 Dependencies.py:226(__init__)
239 0.011 0.000 0.011 0.000 {built-in method posix.rmdir}
3064 0.011 0.000 0.065 0.000 clean.py:25(_remove)
9476 0.011 0.000 0.011 0.000 {method 'tell' of '_io.BufferedRandom' objects}
4737 0.010 0.000 0.125 0.000 zipfile.py:1478(open)
1 0.010 0.010 1.109 1.109 file_finder_git.py:51(_git_interpret_archive)
140 0.010 0.000 0.014 0.000 warnings.py:165(simplefilter)
13153 0.010 0.000 0.014 0.000 <frozen importlib._bootstrap_external>:128(<listcomp>)
6623 0.010 0.000 0.042 0.000 _collections_abc.py:816(get)
279 0.010 0.000 0.359 0.001 subprocess.py:1202(wait)
1067 0.010 0.000 0.030 0.000 fnmatch.py:54(filter)
13153 0.010 0.000 0.027 0.000 <frozen importlib._bootstrap_external>:126(_path_join)
12816 0.010 0.000 0.016 0.000 macosx_libfile.py:246(read_data)
1 0.010 0.010 0.012 0.012 {method 'writerows' of '_csv.writer' objects}
1214 0.010 0.000 0.076 0.000 Utils.py:239(open_source_file)
11009 0.009 0.000 0.009 0.000 {method 'count' of 'bytes' objects}
10431 0.009 0.000 0.022 0.000 <frozen importlib._bootstrap_external>:132(_path_split)
7116 0.009 0.000 0.018 0.000 __init__.py:108(read)
2566 0.009 0.000 0.099 0.000 __init__.py:919(read_text)
11396 0.009 0.000 0.009 0.000 {method 'copy' of 'dict' objects}
3617 0.009 0.000 0.140 0.000 Dependencies.py:454(resolve_depend)
1 0.009 0.009 7.112 7.112 Dependencies.py:749(create_extension_list)
11775 0.009 0.000 0.142 0.000 glob.py:53(_iglob)
11008/11007 0.009 0.000 0.066 0.000 tarfile.py:1138(_proc_member)
11748 0.009 0.000 0.009 0.000 {method 'tell' of '_io.BufferedReader' objects}
81425 0.008 0.000 0.008 0.000 {built-in method _stat.S_ISLNK}
5491 0.008 0.000 0.689 0.000 posixpath.py:391(realpath)
2779 0.008 0.000 0.012 0.000 pathlib.py:56(parse_parts)
9797 0.008 0.000 0.012 0.000 log.py:45(info)
5535 0.008 0.000 0.137 0.000 {method 'extend' of 'list' objects}
24048 0.008 0.000 0.013 0.000 unicode_utils.py:37(try_encode)
14211 0.008 0.000 0.012 0.000 zipfile.py:455(_encodeFilenameFlags)
5487 0.008 0.000 0.054 0.000 file_finder.py:28(_link_not_in_scm)
66 0.008 0.000 0.627 0.010 pkgconfig.py:116(exists)
279 0.008 0.000 0.349 0.001 subprocess.py:1909(_wait)
140 0.008 0.000 0.338 0.002 selectors.py:403(select)
8696 0.007 0.000 0.012 0.000 os.py:754(encode)
2 0.007 0.004 1.357 0.679 env.py:415(cython_aliases)
2 0.007 0.004 0.197 0.098 macosx_libfile.py:359(calculate_macosx_platform_tag)
28422 0.007 0.000 0.007 0.000 zipfile.py:1116(_fileobj)
28458 0.007 0.000 0.010 0.000 unicode_utils.py:18(filesys_decode)
4737 0.007 0.000 0.009 0.000 zipfile.py:1704(_writecheck)
3269 0.007 0.000 0.007 0.000 {built-in method _codecs.utf_8_decode}
13603 0.007 0.000 0.007 0.000 re.py:324(_subx)
43/34 0.007 0.000 0.044 0.001 {built-in method _imp.exec_dynamic}
441/145 0.007 0.000 0.018 0.000 sre_parse.py:494(_parse)
1067 0.007 0.000 0.065 0.000 glob.py:162(_listdir)
23054 0.007 0.000 0.009 0.000 posixpath.py:52(normcase)
8928 0.007 0.000 0.007 0.000 filelist.py:44(debug_print)
151 0.007 0.000 0.009 0.000 contextlib.py:496(callback)
142 0.007 0.000 0.008 0.000 subprocess.py:1586(_get_handles)
64 0.007 0.000 0.007 0.000 shlex.py:21(__init__)
2233 0.007 0.000 0.010 0.000 tokenize.py:431(_tokenize)
5112 0.007 0.000 0.011 0.000 pathlib.py:621(__str__)
7116 0.007 0.000 0.046 0.000 __init__.py:102(<genexpr>)
534 0.007 0.000 0.102 0.000 sage_build_ext.py:90(prepare_extension)
22617 0.007 0.000 0.010 0.000 glob.py:96(<genexpr>)
7390/355 0.006 0.000 0.039 0.000 signature.py:100(extract_combiner)
24550 0.006 0.000 0.006 0.000 {method 'union' of 'set' objects}
534 0.006 0.000 0.172 0.000 sage_build_cython.py:254(create_extension)
4737 0.006 0.000 0.032 0.000 zipfile.py:1106(__init__)
2785 0.006 0.000 0.071 0.000 dir_util.py:15(mkpath)
656 0.006 0.000 0.037 0.000 <frozen importlib._bootstrap_external>:1536(find_spec)
1 0.006 0.006 7.412 7.412 Dependencies.py:881(cythonize)
7093 0.006 0.000 0.044 0.000 posixpath.py:164(islink)
52846 0.006 0.000 0.006 0.000 {method 'is_dir' of 'posix.DirEntry' objects}
4736 0.006 0.000 0.006 0.000 {method 'digest' of '_hashlib.HASH' objects}
2779 0.006 0.000 0.019 0.000 pathlib.py:569(_parse_args)
142 0.006 0.000 0.006 0.000 subprocess.py:246(_cleanup)
2350 0.006 0.000 0.142 0.000 build_py.py:337(build_module)
4550 0.006 0.000 0.009 0.000 _collections.py:28(parse)
9566 0.006 0.000 0.008 0.000 tarfile.py:1396(_block)
4336 0.006 0.000 0.840 0.000 Dependencies.py:522(parse_dependencies)
534 0.006 0.000 0.008 0.000 Dependencies.py:278(subs)
3500 0.006 0.000 0.098 0.000 Dependencies.py:981(copy_to_build_dir)
25802 0.006 0.000 0.006 0.000 {method 'seek' of '_io.BufferedReader' objects}
3201 0.005 0.000 0.008 0.000 gast.py:17(create_node)
534 0.005 0.000 0.042 0.000 dep_util.py:56(newer_group)
1459/1411 0.005 0.000 0.219 0.000 Dependencies.py:545(cimports_externs_incdirs)
10318/10316 0.005 0.000 0.005 0.000 {method 'format' of 'str' objects}
138 0.005 0.000 0.050 0.000 os.py:619(get_exec_path)
7116 0.005 0.000 0.058 0.000 __init__.py:381(<genexpr>)
4736 0.005 0.000 0.020 0.000 util.py:26(urlsafe_b64encode)
142 0.005 0.000 0.006 0.000 contextlib.py:533(__exit__)
2136 0.005 0.000 0.013 0.000 macosx_libfile.py:223(get_base_class_and_magic_number)
20862 0.005 0.000 0.007 0.000 <frozen importlib._bootstrap_external>:134(<genexpr>)
1 0.005 0.005 0.054 0.054 filelist.py:61(sort)
Some interesting ones:
1 0.000 0.000 4.102 4.102 egg_info.py:309(find_sources)
1 0.000 0.000 1.648 1.648 sdist.py:349(read_template)
1 0.000 0.000 4.151 4.151 install_scripts.py:18(run)
1 0.010 0.010 1.109 1.109 file_finder_git.py:51(_git_interpret_archive)
1 0.000 0.000 1.346 1.346 filelist.py:41(findall)
mkoeppe commented
comment:32
file_finder_git leaks in because setuptools_scm is installed in site-packages. pypa/setuptools-scm#561, pypa/setuptools#2652
Workaround: pypa/setuptools-scm#190 (comment)
sagetrac-git commented
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
dcb5c79 | pkgs/sagemath-standard/setup.py: Disable setuptools_scm file-finder |
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
62be88a | src/MANIFEST.in: Do not use global-include |
sagetrac-git commented
mkoeppe commented
comment:35
sage -t --random-seed=307782328559689550615012446878769850043 sage_setup/clean.py
**********************************************************************
File "sage_setup/clean.py", line 98, in sage_setup.clean._find_stale_files
Failed example:
for f in stale_iter:
if f.endswith(skip_extensions): continue
if '/ext_data/' in f: continue
print('Found stale file: ' + f)
Expected nothing
Got:
Found stale file: sage/libs/linkages/padics/__init__.py
Found stale file: sage/libs/linkages/padics/relaxed/__init__.py
Found stale file: sage/tests/books/judson-abstract-algebra/struct-sage.py
Found stale file: sage/tests/books/judson-abstract-algebra/actions-sage.py
Found stale file: sage/tests/books/judson-abstract-algebra/normal-sage.py
Found stale file: sage/tests/books/judson-abstract-algebra/boolean-sage.py
...
kwankyu commented
comment:36
I cannot reproduce the failure.
What does it mean? Those files are not stale files. As far as I understand, there can be no stale files in the wheel install. So the failure just indicates that find_python_sources is buggy?
mkoeppe commented
comment:37
This branch removes cleaning of the installation directory (site-packages) but keeps the cleaning of the build directory:
--- a/src/sage_setup/command/sage_install.py
+++ b/src/sage_setup/command/sage_install.py
@@ -93,9 +93,8 @@ class sage_clean(install):
nobase_data_files = [(src_dir, [os.path.join(src_dir, filename) for filename in filenames])
for package, src_dir, build_dir, filenames in cmd_build_py.data_files]
- # Clean install directory (usually, purelib and platlib are the same)
- # and build directory.
- output_dirs = [self.install_purelib, self.install_platlib, self.build_lib]
+ # Clean build directory.
+ output_dirs = [self.build_lib]
from sage_setup.clean import clean_install_dir
for output_dir in set(output_dirs):
log.info('- cleaning {0}'.format(output_dir))
So there is still a notion of stale files.
You are right that this doctest (in _find_stale_files) is outdated. It tests SAGE_LIB, i.e., the installation directory.
I agree that this points to a bug in find_python_sources in this branch. I'll investigate
mkoeppe commented
comment:38
These files from comment:35 are definitely in the wheel.
mkoeppe commented
comment:39
I can reproduce the failure locally with ./configure --enable-editable.
mkoeppe commented
comment:40
In this mode, of course SAGE_LIB is the same as SAGE_SRC, which makes that whole doctest meaningless
mkoeppe commented
comment:41
src/sage/tests/books is not a package (no __init__.py) or namespace package according to our convention (no all*.py file). So it's correct that it does not show up in find_python_sources.
I'll just remove the outdated doctest.
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
ab5f37e | /Users/mkoeppe/s/sage/sage-rebasing/worktree-rebase/src/sage/libs/linkages/__init__.py: New |
sagetrac-git commented
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
15ef369 | src/sage_setup/clean.py: Update _find_stale_files doctest to use importlib.metadata.files('sagemath-standard') |
sagetrac-git commented
mkoeppe commented
comment:44
I found a better solution. Passes all tests for me now.
mkoeppe commented
comment:46
Replying to Matthias Köppe:
It would be worth checking if we can use an on-disk cache to speed up dependency tracking and package/module discovery.
I've opened #34630 (Switch sagelib's build system from setuptools to meson-python / ninja) for this
kwankyu commented
comment:47
On my machine, sage -b for a trivial change spends 50% more time in disable-editable mode (about 45 seconds) than in enable-editable mode (about 30 seconds).
Personally I am okay with this as I use the default enable-editable mode.
For people using disable-editable mode for development, it may be painful to accept 50% more build time for trivial changes.
I think we should warn developers in sage-devel for this change and establish enable-editable as the recommended mode for sage development.
mkoeppe commented
comment:48
I'll make this change depend on a configure option for now. (See changed ticket description.)
mkoeppe commented
Description changed:
---
+++
@@ -1,9 +1,15 @@
When `./configure --disable-editable` is in use (the default before #32406), the Sage library is installed using its custom incremental implementation of `setup.py install` -- which installs its files in `site-packages` and then removes the files that it does not know about (the "cleaner").
-The cleaner is incompatible with namespace packages and trying to fix it would lead to complicated and awkward code (#32927). (Besides, `setup.py install` is deprecated since https://setuptools.pypa.io/en/latest/history.html#v58-3-0)
+Problem 1: The cleaner is incompatible with namespace packages and trying to fix it would lead to complicated and awkward code (#32927). (Besides, `setup.py install` is deprecated since https://setuptools.pypa.io/en/latest/history.html#v58-3-0)
-In this ticket, we change the installation procedure to go through wheel building and installation - like we do with all other Python packages.
+Problem 2: Because of the direct installation, we do not have wheels for sagemath-standard available (even if `make wheels` is used explicitly). Such wheels can be useful for making separate venvs.
-However, this makes a trivial `./sage -b` much slower (from 28s as per timings in [#32406 comment:21](https://github.com/sagemath/sage/issues/32406#comment:21) to 70s). This should be acceptable because after #32406, the primary use case of `./configure --disable-editable` is for fixed installations in an installation prefix, not for development.
+In this ticket:
+- We create a new option `./configure --enable-sage-wheels` (which can be used both with `./configure --enable-editable` and `./configure --disable-editable`).
+- We remove the use of the installation-dir cleaner.
+- If enabled, installation of sagelib will go through wheel building and installation - like we do with all other Python packages.
+- If disabled, the only change is that we uninstall sagelib before installing it.
+A trivial `./sage -b` is much slower with `./configure --disable-editable --enable-sage-wheels` compared to `./configure --disabled-editable` (from 28s as per timings in [#32406 comment:21](https://github.com/sagemath/sage/issues/32406#comment:21) to 70s).
+This ticket brings a number of speed improvements to mitigate this.
sagetrac-git commented
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
bb0c3ec | configure.ac (--enable-wheels): New |
mkoeppe commented
Description changed:
---
+++
@@ -5,11 +5,11 @@
Problem 2: Because of the direct installation, we do not have wheels for sagemath-standard available (even if `make wheels` is used explicitly). Such wheels can be useful for making separate venvs.
In this ticket:
-- We create a new option `./configure --enable-sage-wheels` (which can be used both with `./configure --enable-editable` and `./configure --disable-editable`).
+- We create a new option `./configure --enable-wheels` (which can be used both with `./configure --enable-editable` and `./configure --disable-editable`).
- We remove the use of the installation-dir cleaner.
- If enabled, installation of sagelib will go through wheel building and installation - like we do with all other Python packages.
- If disabled, the only change is that we uninstall sagelib before installing it.
-A trivial `./sage -b` is much slower with `./configure --disable-editable --enable-sage-wheels` compared to `./configure --disabled-editable` (from 28s as per timings in [#32406 comment:21](https://github.com/sagemath/sage/issues/32406#comment:21) to 70s).
+A trivial `./sage -b` is much slower with `./configure --disable-editable --enable-wheels` compared to `./configure --disabled-editable` (from 28s as per timings in [#32406 comment:21](https://github.com/sagemath/sage/issues/32406#comment:21) to 70s).
This ticket brings a number of speed improvements to mitigate this.
mkoeppe commented
Description changed:
---
+++
@@ -8,7 +8,7 @@
- We create a new option `./configure --enable-wheels` (which can be used both with `./configure --enable-editable` and `./configure --disable-editable`).
- We remove the use of the installation-dir cleaner.
- If enabled, installation of sagelib will go through wheel building and installation - like we do with all other Python packages.
-- If disabled, the only change is that we uninstall sagelib before installing it.
+- If disabled, the only change is that we uninstall sagelib before installing it (that's the replacement for the installation-dir cleaner).
A trivial `./sage -b` is much slower with `./configure --disable-editable --enable-wheels` compared to `./configure --disabled-editable` (from 28s as per timings in [#32406 comment:21](https://github.com/sagemath/sage/issues/32406#comment:21) to 70s).
This ticket brings a number of speed improvements to mitigate this.sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
6ff338b | configure.ac: Improve docstring |
sagetrac-git commented
mkoeppe commented
Description changed:
---
+++
@@ -10,6 +10,6 @@
- If enabled, installation of sagelib will go through wheel building and installation - like we do with all other Python packages.
- If disabled, the only change is that we uninstall sagelib before installing it (that's the replacement for the installation-dir cleaner).
-A trivial `./sage -b` is much slower with `./configure --disable-editable --enable-wheels` compared to `./configure --disabled-editable` (from 28s as per timings in [#32406 comment:21](https://github.com/sagemath/sage/issues/32406#comment:21) to 70s).
+A trivial `./sage -b` is much slower with `./configure --disable-editable --enable-wheels` compared to `./configure --disable-editable` (from 28s as per timings in [#32406 comment:21](https://github.com/sagemath/sage/issues/32406#comment:21) to 70s).
This ticket brings a number of speed improvements to mitigate this.
sagetrac-git commented
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
c6acede | build/pkgs/{sage_conf,sage_docbuild,sage_setup,sage_sws2rst}/spkg-install: Handle SAGE_WHEELS |
sagetrac-git commented
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
82d4c87 | build/pkgs/sagelib/spkg-install: Only uninstall old sagelib after successful build |
mkoeppe commented
Description changed:
---
+++
@@ -8,7 +8,7 @@
- We create a new option `./configure --enable-wheels` (which can be used both with `./configure --enable-editable` and `./configure --disable-editable`).
- We remove the use of the installation-dir cleaner.
- If enabled, installation of sagelib will go through wheel building and installation - like we do with all other Python packages.
-- If disabled, the only change is that we uninstall sagelib before installing it (that's the replacement for the installation-dir cleaner).
+- If disabled, the only change is that after a successful build, we uninstall sagelib before installing it (that's the replacement for the installation-dir cleaner).
A trivial `./sage -b` is much slower with `./configure --disable-editable --enable-wheels` compared to `./configure --disable-editable` (from 28s as per timings in [#32406 comment:21](https://github.com/sagemath/sage/issues/32406#comment:21) to 70s).
This ticket brings a number of speed improvements to mitigate this.sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
6e6cd7d | src/sage/misc/package_dir.py: Fix docstring markup |
sagetrac-git commented
sagetrac-git commented
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
6374524 | pkgs/sagemath-standard/setup.py: Disable setuptools_scm file-finder |
a7c4e66 | src/MANIFEST.in: Do not use global-include |
3f7747c | /Users/mkoeppe/s/sage/sage-rebasing/worktree-rebase/src/sage/libs/linkages/__init__.py: New |
e66952e | src/sage_setup/clean.py: Update _find_stale_files doctest to use importlib.metadata.files('sagemath-standard') |
19171cf | configure.ac (--enable-wheels): New |
adb1848 | configure.ac: Improve docstring |
cb3caa9 | build/pkgs/{sage_conf,sage_docbuild,sage_setup,sage_sws2rst}/spkg-install: Handle SAGE_WHEELS |
7e13d1c | build/pkgs/sagelib/spkg-install: Only uninstall old sagelib after successful build |
6a5ed77 | src/sage/misc/package_dir.py: Fix docstring markup |
e9ef5e5 | is_package_or_sage_namespace_package_dir: Add option distribution_filter |
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
e91a0bd | pkgs/sagemath-standard/setup.py: Disable setuptools_scm file-finder |
e2fbdc8 | src/MANIFEST.in: Do not use global-include |
ef46de5 | /Users/mkoeppe/s/sage/sage-rebasing/worktree-rebase/src/sage/libs/linkages/__init__.py: New |
67ff945 | src/sage_setup/clean.py: Update _find_stale_files doctest to use importlib.metadata.files('sagemath-standard') |
a683934 | configure.ac (--enable-wheels): New |
f2d1576 | configure.ac: Improve docstring |
05b44d7 | build/pkgs/{sage_conf,sage_docbuild,sage_setup,sage_sws2rst}/spkg-install: Handle SAGE_WHEELS |
dde4bdb | src/sage/misc/package_dir.py: Fix docstring markup |
ffe67e8 | is_package_or_sage_namespace_package_dir: Add option distribution_filter |
11507a5 | build/pkgs/sagelib/spkg-install: No need to uninstall before installing - cleaner is back in |
sagetrac-git commented
mkoeppe commented
comment:60
Reduced the scope of the ticket to make it easier to review.
I'm postponing the changes to the cleaner to another ticket.
mkoeppe commented
Description changed:
---
+++
@@ -6,9 +6,8 @@
In this ticket:
- We create a new option `./configure --enable-wheels` (which can be used both with `./configure --enable-editable` and `./configure --disable-editable`).
-- We remove the use of the installation-dir cleaner.
- If enabled, installation of sagelib will go through wheel building and installation - like we do with all other Python packages.
-- If disabled, the only change is that after a successful build, we uninstall sagelib before installing it (that's the replacement for the installation-dir cleaner).
+- If disabled, there is no change.
A trivial `./sage -b` is much slower with `./configure --disable-editable --enable-wheels` compared to `./configure --disable-editable` (from 28s as per timings in [#32406 comment:21](https://github.com/sagemath/sage/issues/32406#comment:21) to 70s).
This ticket brings a number of speed improvements to mitigate this.mkoeppe commented
Description changed:
---
+++
@@ -2,11 +2,11 @@
Problem 1: The cleaner is incompatible with namespace packages and trying to fix it would lead to complicated and awkward code (#32927). (Besides, `setup.py install` is deprecated since https://setuptools.pypa.io/en/latest/history.html#v58-3-0)
-Problem 2: Because of the direct installation, we do not have wheels for sagemath-standard available (even if `make wheels` is used explicitly). Such wheels can be useful for making separate venvs.
+Problem 2: Because of the direct installation, we do not have wheels for sagemath-standard available (even if [make wheels](https://trac.sagemath.org/wiki/ReleaseTours/sage-9.7#Editableinstallationisnowthedefault) is used explicitly). Such wheels can be useful for making separate venvs.
In this ticket:
- We create a new option `./configure --enable-wheels` (which can be used both with `./configure --enable-editable` and `./configure --disable-editable`).
-- If enabled, installation of sagelib will go through wheel building and installation - like we do with all other Python packages.
+- If enabled, installation of sagelib will go through wheel building and installation - like we do with all other Python packages. This enables modularized builds (#34587).
- If disabled, there is no change.
A trivial `./sage -b` is much slower with `./configure --disable-editable --enable-wheels` compared to `./configure --disable-editable` (from 28s as per timings in [#32406 comment:21](https://github.com/sagemath/sage/issues/32406#comment:21) to 70s).kwankyu commented
comment:63
./configure --enable-wheels doesn't seem to work for me, even after ./bootstrap (is this necessary?)
kwankyu commented
comment:64
Replying to Kwankyu Lee:
./configure --enable-wheelsdoesn't seem to work for me, even after./bootstrap(is this necessary?)
sage -b after ./configure --disable-editable does not install wheel. sage -b just takes 20 seconds.
mkoeppe commented
comment:65
Yes, there are 4 modes of operation now, all combinations of --{en,dis}able-editable and --{en,dis}able-wheels should work
kwankyu commented
comment:66
What doesn't work is to configure those options separately: ./configure --enable-wheels and then ./configure --disable-editable. I guess this is normal.
Yes, ./configure --disable-editable --enable-wheels does build the wheel :)
mkoeppe commented
comment:67
To add options to the previous list of configure options, there's unfortunately no shorter idiom than
eval ./configure $(./config.status --config) NEW-OPTIONS...
kwankyu commented
comment:68
Making --enable-wheels default eventually is necessary for sagelib modularization. Right?
mkoeppe commented
comment:69
I'd say it will be necessary as soon as the first external packages arrive that declare a dependency on a modularized distribution rather than on sagemath-standard.
Until then we can get away with keeping both --enable-editable and --disable-editable --disable-wheels monolithic; and we can use --disable-editable --enable-wheels for working on the details of the modularized distributions.
kwankyu commented
comment:71
Questions on changes in src/MANIFEST.in:
-
Is this line
include MANIFEST.innecessary? Perhaps becausesrc/MANIFEST.inis a symlink? -
How was this list made ? Could the hard-coded list be still valid in future?
include sage/modular/arithgroup/farey_symbol.h
include sage/cpython/debugimpl.c
include sage/graphs/base/boost_interface.cpp
include sage/graphs/cliquer/cl.c
include sage/graphs/graph_decompositions/sage_tdlib.cpp
include sage/libs/eclib/wrap.cpp
include sage/libs/linkages/padics/relaxed/flint_helper.c
include sage/misc/inherit_comparison_impl.c
include sage/modular/arithgroup/farey.cpp
include sage/modular/arithgroup/sl2z.cpp
include sage/rings/bernmm/bern_modp.cpp
include sage/rings/bernmm/bern_modp_util.cpp
include sage/rings/bernmm/bern_rat.cpp
include sage/rings/bernmm/bernmm-test.cpp
include sage/rings/padics/transcendantal.c
include sage/rings/polynomial/weil/power_sums.c
include sage/schemes/hyperelliptic_curves/hypellfrob/hypellfrob.cpp
include sage/schemes/hyperelliptic_curves/hypellfrob/recurrences_ntl.cpp
include sage/schemes/hyperelliptic_curves/hypellfrob/recurrences_zn_poly.cpp
include sage/stats/distributions/dgs_bern.c
include sage/stats/distributions/dgs_gauss_dp.c
include sage/stats/distributions/dgs_gauss_mp.c
include sage/symbolic/ginac/*.cpp
kwankyu commented
Reviewer: Kwankyu Lee
sagetrac-git commented
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
974c7fd | src/MANIFEST.in: Remove redundant entries for MANIFEST.in, pyproject.toml |
mkoeppe commented
comment:74
Replying to Kwankyu Lee:
Questions on changes in
src/MANIFEST.in:
- Is this line
include MANIFEST.innecessary? Perhaps becausesrc/MANIFEST.inis a symlink?
No, not necessary any more; it probably was necessary with prehistoric versions of setuptools.
sagetrac-git commented
Branch pushed to git repo; I updated commit sha1. New commits:
25898ba | src/MANIFEST.in: Exclude generated file farey_symbol.h; add comments |
sagetrac-git commented
mkoeppe commented
comment:76
Replying to Kwankyu Lee:
- How was this list made ?
I've added a comment (and fixed a mistake that I had made along the way).
kwankyu commented
comment:77
Thanks.
It works well and looks good to me.
mkoeppe commented
comment:78
Thank you!
vbraun commented
Changed branch from u/mkoeppe/remove_use_of__setup_py_install__for_sagelib to 25898ba
antonio-rojas commented
Changed commit from 25898ba to none
antonio-rojas commented
comment:80
This makes tests depend on sage_setup, so one gets failures when testing an installed sage
sage -t --long --random-seed=334405888414880658562662015982432628863 /usr/lib/python3.10/site-packages/sage/misc/package_dir.py
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/misc/package_dir.py", line 108, in sage.misc.package_dir.read_distribution
Failed example:
from sage_setup.find import read_distribution
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
exec(compiled, globs)
File "<doctest sage.misc.package_dir.read_distribution[1]>", line 1, in <module>
from sage_setup.find import read_distribution
ModuleNotFoundError: No module named 'sage_setup'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/misc/package_dir.py", line 109, in sage.misc.package_dir.read_distribution
Failed example:
read_distribution(os.path.join(SAGE_SRC, 'sage', 'graphs', 'graph_decompositions', 'tdlib.pyx'))
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
exec(compiled, globs)
File "<doctest sage.misc.package_dir.read_distribution[2]>", line 1, in <module>
read_distribution(os.path.join(SAGE_SRC, 'sage', 'graphs', 'graph_decompositions', 'tdlib.pyx'))
NameError: name 'read_distribution' is not defined
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/misc/package_dir.py", line 111, in sage.misc.package_dir.read_distribution
Failed example:
read_distribution(os.path.join(SAGE_SRC, 'sage', 'graphs', 'graph_decompositions', 'modular_decomposition.py'))
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
exec(compiled, globs)
File "<doctest sage.misc.package_dir.read_distribution[3]>", line 1, in <module>
read_distribution(os.path.join(SAGE_SRC, 'sage', 'graphs', 'graph_decompositions', 'modular_decomposition.py'))
NameError: name 'read_distribution' is not defined
**********************************************************************