Replace use of module_list and OptionalExtension by extending find_python_sources

Question

Replace use of module_list and OptionalExtension by extending find_python_sources

Closed this issue 5 years ago · 87 comments

We add two new features to find_python_sources: finding Cython extensions, and filtering by "distributions".

We remove the use of module_list, finding Cython extensions instead in the source tree. (This is prepared by #29706 and follow-up tickets by moving Extension options to directives in the source files.)

We remove OptionalExtensions as follows. We map installed packages to "distributions" (for example, tdlib -> sage-tdlib) and then filter by distribution.

Follow-up tickets:

as part of Meta-ticket #29705, we will make these "distributions" actually separate distutils packages that provide namespace packages (#28925)
remove the file module_list.py (not done on this ticket to avoid possible merge conflicts), deprecate OptionalExtension (which may be in use by user packages?)

CC: @kiwifb @isuruf @videlec @dcoudert @dimpase @kliem @vbraun

Component: refactoring

Author: Matthias Koeppe

Branch/Commit: 55c3fbc

Reviewer: Jonathan Kliem

Issue created by migration from https://trac.sagemath.org/ticket/29701

Answer 1 · 2020-05-17T17:30:34.000Z

Description changed:

--- 
+++ 
@@ -1 +1,4 @@
+Preparation:
+- #28925 Modify clean_stale_files to support modularization of sagelib by namespace packages
 
+

Answer 2 · 2020-05-17T17:40:02.000Z

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
 Preparation:
 - #28925 Modify clean_stale_files to support modularization of sagelib by namespace packages
+- #29702 Move all code from src/setup.py to sage_setup
 
-

Answer 3 · 2020-05-17T19:54:25.000Z

Description changed:

--- 
+++ 
@@ -1,4 +1,16 @@
 Preparation:
-- #28925 Modify clean_stale_files to support modularization of sagelib by namespace packages
-- #29702 Move all code from src/setup.py to sage_setup
+- #28925 Modify `clean_stale_files` to support modularization of `sagelib` by namespace packages
+- #29702 Move all code from `src/setup.py` to `sage_setup`
 
+Tickets for individual `OptionalExtension`s (see `src/module_list.py`):
+- `sage.graphs.mcqd`
+- `sage.graphs.bliss`
+- `sage.graphs.graph_decompositions.tdlib`
+- `sage.interfaces.primecount` (how come this is not in `sage.libs`, where `primecount.pxd` lives?)
+- `sage.libs.coxeter3.coxeter`
+- `sage.libs.fes`
+- `sage.libs.sirocco`
+- `sage.libs.meataxe`, `sage.matrix.matrix_gfpn_dense`
+
+
+

Answer 4 · 2020-05-17T21:44:30.000Z

Description changed:

--- 
+++ 
@@ -5,7 +5,7 @@
 Tickets for individual `OptionalExtension`s (see `src/module_list.py`):
 - `sage.graphs.mcqd`
 - `sage.graphs.bliss`
-- `sage.graphs.graph_decompositions.tdlib`
+- `sage.graphs.graph_decompositions.tdlib` (used as an example in #28925)
 - `sage.interfaces.primecount` (how come this is not in `sage.libs`, where `primecount.pxd` lives?)
 - `sage.libs.coxeter3.coxeter`
 - `sage.libs.fes`

Answer 5 · 2020-05-18T00:18:04.000Z

Description changed:

--- 
+++ 
@@ -1,5 +1,6 @@
 Preparation:
 - #28925 Modify `clean_stale_files` to support modularization of `sagelib` by namespace packages
+- Remove empty  `__init__.py` files to convert packages to native namespace packages
 - #29702 Move all code from `src/setup.py` to `sage_setup`
 
 Tickets for individual `OptionalExtension`s (see `src/module_list.py`):

Answer 6 · 2020-05-19T04:41:09.000Z

Description changed:

--- 
+++ 
@@ -1,5 +1,9 @@
+Currently, a user would install, for example, the optional package `tdlib` and then rebuild `sagelib` so that the `OptionalExtension` `sage.graphs.graph_decompositions.tdlib` is built and installed.
+
+With this ticket, the user would instead install a new optional package `sage_tdlib` (which has `tdlib` as a dependency); this installs the `Extension` `sage.graphs.graph_decompositions.tdlib` (as a namespace package).
+
 Preparation:
-- #28925 Modify `clean_stale_files` to support modularization of `sagelib` by namespace packages
+- #28925 Modify find_python_sources, clean_stale_files to support modularization of sagelib by native namespace packages (PEP 420)
 - Remove empty  `__init__.py` files to convert packages to native namespace packages
 - #29702 Move all code from `src/setup.py` to `sage_setup`

Answer 7 · 2020-05-19T04:41:52.000Z

Description changed:

--- 
+++ 
@@ -3,7 +3,7 @@
 With this ticket, the user would instead install a new optional package `sage_tdlib` (which has `tdlib` as a dependency); this installs the `Extension` `sage.graphs.graph_decompositions.tdlib` (as a namespace package).
 
 Preparation:
-- #28925 Modify find_python_sources, clean_stale_files to support modularization of sagelib by native namespace packages (PEP 420)
+- #28925 Modify `find_python_sources`, `clean_stale_files` to support modularization of sagelib by native namespace packages (PEP 420)
 - Remove empty  `__init__.py` files to convert packages to native namespace packages
 - #29702 Move all code from `src/setup.py` to `sage_setup`

Answer 8 · 2020-05-20T22:54:18.000Z

Description changed:

--- 
+++ 
@@ -4,6 +4,7 @@
 
 Preparation:
 - #28925 Modify `find_python_sources`, `clean_stale_files` to support modularization of sagelib by native namespace packages (PEP 420)
+- Map installed packages/features to virtual `sage_packages` (for example, `tdlib` -> `sage_tdlib`), replace `module_list.py` and the `OptionalExtension` mechanism by passing `sage_packages` to `find_python_sources`.
 - Remove empty  `__init__.py` files to convert packages to native namespace packages
 - #29702 Move all code from `src/setup.py` to `sage_setup`

Answer 9 · 2020-06-03T18:56:13.000Z

Description changed:

--- 
+++ 
@@ -4,7 +4,7 @@
 
 Preparation:
 - #28925 Modify `find_python_sources`, `clean_stale_files` to support modularization of sagelib by native namespace packages (PEP 420)
-- Map installed packages/features to virtual `sage_packages` (for example, `tdlib` -> `sage_tdlib`), replace `module_list.py` and the `OptionalExtension` mechanism by passing `sage_packages` to `find_python_sources`.
+- Map installed packages/features to virtual `distributions` (for example, `tdlib` -> `sage-tdlib`), replace `module_list.py` and the `OptionalExtension` mechanism by passing `distributions` to `find_python_sources`.
 - Remove empty  `__init__.py` files to convert packages to native namespace packages
 - #29702 Move all code from `src/setup.py` to `sage_setup`

Answer 10 · 2020-06-03T22:09:56.000Z

Dependencies: #29411, #29702, #29706, #29720, #29721, #29785, #29786, #29790, #29791, #28925

Answer 11 · 2020-06-03T22:09:56.000Z

Description changed:

--- 
+++ 
@@ -1,22 +1,7 @@
-Currently, a user would install, for example, the optional package `tdlib` and then rebuild `sagelib` so that the `OptionalExtension` `sage.graphs.graph_decompositions.tdlib` is built and installed.
+Using the extended `find_python_sources` from #28925, 
+we remove the use of `module_list`, finding Cython extensions instead in the source tree.  (This is prepared by #29706 and follow-up tickets by moving Extension options to directives in the source files.)
 
-With this ticket, the user would instead install a new optional package `sage_tdlib` (which has `tdlib` as a dependency); this installs the `Extension` `sage.graphs.graph_decompositions.tdlib` (as a namespace package).
+We remove `OptionalExtension`s as follows. We map installed packages to "distributions" (for example, `tdlib` -> `sage-tdlib`) and then filter by distribution.
 
-Preparation:
-- #28925 Modify `find_python_sources`, `clean_stale_files` to support modularization of sagelib by native namespace packages (PEP 420)
-- Map installed packages/features to virtual `distributions` (for example, `tdlib` -> `sage-tdlib`), replace `module_list.py` and the `OptionalExtension` mechanism by passing `distributions` to `find_python_sources`.
-- Remove empty  `__init__.py` files to convert packages to native namespace packages
-- #29702 Move all code from `src/setup.py` to `sage_setup`
+(In a follow-up ticket, part of Meta-ticket #29705, we will make these "distributions" actually separate distutils packages.)
 
-Tickets for individual `OptionalExtension`s (see `src/module_list.py`):
-- `sage.graphs.mcqd`
-- `sage.graphs.bliss`
-- `sage.graphs.graph_decompositions.tdlib` (used as an example in #28925)
-- `sage.interfaces.primecount` (how come this is not in `sage.libs`, where `primecount.pxd` lives?)
-- `sage.libs.coxeter3.coxeter`
-- `sage.libs.fes`
-- `sage.libs.sirocco`
-- `sage.libs.meataxe`, `sage.matrix.matrix_gfpn_dense`
-
-
-

Answer 12 · 2020-06-03T22:10:32.000Z

Branch: u/mkoeppe/replace_use_of_module_list_optionalextension

Answer 13 · 2020-06-03T22:48:38.000Z

Commit: 891d12a

Answer 14 · 2020-06-03T22:48:38.000Z

Last 10 new commits:

`4365e5d`	`Merge branch 't/29790/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_5__sage_graphs_' into t/29705/META-modularize-sagelib`
`9dc7022`	`Merge branch 't/29706/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files' into t/29791/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_6__last_`
`f78b06d`	`src/module_list.py: Move options for Extensions in sage.libs to distutils directives`
`1b0e29d`	`src/module_list.py: Move options for Extensions in sage.matrix to distutils directives`
`6421e2c`	`src/module_list.py: Move remaining options for Extensions in sage.libs, sage.rings to distutils directives`
`b3d3d2f`	`Merge branch 't/29791/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_6__last_' into t/29705/META-modularize-sagelib`
`2821934`	`Fix sage_setup directives: Use distribution, not package`
`9052db4`	`Merge branch 't/29720/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_2___optionalextensions_' into t/29705/META-modularize-sagelib`
`ff710ee`	`src/sage_setup/optional_extension.py (is_package_installed_and_updated): Factor out from OptionalExtension`
`891d12a`	`src/setup.py: Remove use of module_list.py; filter by distributions`

Answer 15 · 2020-06-03T22:48:38.000Z

Description changed:

--- 
+++ 
@@ -1,5 +1,6 @@
-Using the extended `find_python_sources` from #28925, 
-we remove the use of `module_list`, finding Cython extensions instead in the source tree.  (This is prepared by #29706 and follow-up tickets by moving Extension options to directives in the source files.)
+We add two new features to `find_python_sources`: finding Cython extensions, and filtering by "distributions".
+ 
+We remove the use of `module_list`, finding Cython extensions instead in the source tree.  (This is prepared by #29706 and follow-up tickets by moving Extension options to directives in the source files.)
 
 We remove `OptionalExtension`s as follows. We map installed packages to "distributions" (for example, `tdlib` -> `sage-tdlib`) and then filter by distribution.

Answer 16 · 2020-06-03T22:48:38.000Z

Changed dependencies from #29411, #29702, #29706, #29720, #29721, #29785, #29786, #29790, #29791, #28925 to #29411, #29702, #29706, #29720, #29721, #29785, #29786, #29790, #29791

Answer 17 · 2020-06-03T22:50:55.000Z

Changed dependencies from #29411, #29702, #29706, #29720, #29721, #29785, #29786, #29790, #29791 to #29702 (#29411), #29706, #29720, #29721, #29785, #29786, #29790, #29791

Answer 18 · 2020-06-03T22:53:24.000Z

Changed commit from 891d12a to ec7e9c5

Answer 19 · 2020-06-03T22:53:24.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`0295c8f`	`src/module_list.py: Move options for Extensions in sage.graphs.graph_decompositions to distutils directives`
`b582789`	`src/sage_setup/find.py: Filter by directive 'sage_setup: distribution = PKG', find Cython modules`
`63c64d5`	Merge branches 't/29720/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_2___optionalextensions_', 't/29721/spkg_configure_m4_for_coxeter3', 't/29785/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_3__get_rid_of_uname_specific_', 't/29786/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_4__sage_rings_', 't/29790/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_5__sage_graphs_' and 't/29791/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_6__last_' into t/29701/replace_use_of_module_list_optionalextension
`ae70c81`	`src/sage_setup/optional_extension.py (is_package_installed_and_updated): Factor out from OptionalExtension`
`ec7e9c5`	`src/setup.py: Remove use of module_list.py; filter by distributions`

Answer 20 · 2020-06-03T22:54:40.000Z

comment:15

No longer on top of #28925, so no namespace package are involved any more for this ticket.

Answer 21 · 2020-06-03T22:56:38.000Z

Description changed:

--- 
+++ 
@@ -4,5 +4,9 @@
 
 We remove `OptionalExtension`s as follows. We map installed packages to "distributions" (for example, `tdlib` -> `sage-tdlib`) and then filter by distribution.
 
-(In a follow-up ticket, part of Meta-ticket #29705, we will make these "distributions" actually separate distutils packages.)
+Follow-up tickets:
+- as part of Meta-ticket #29705, we will make these "distributions" actually separate distutils packages
+- remove the file `module_list.py` (not done on this ticket to avoid possible merge conflicts), deprecate `OptionalExtension` (which may be in use by user packages?)
 
+
+

Answer 22 · 2020-06-03T22:56:38.000Z

Author: Matthias Koeppe

Answer 23 · 2020-06-03T22:57:15.000Z

Description changed:

--- 
+++ 
@@ -1,5 +1,5 @@
 We add two new features to `find_python_sources`: finding Cython extensions, and filtering by "distributions".
- 
+
 We remove the use of `module_list`, finding Cython extensions instead in the source tree.  (This is prepared by #29706 and follow-up tickets by moving Extension options to directives in the source files.)
 
 We remove `OptionalExtension`s as follows. We map installed packages to "distributions" (for example, `tdlib` -> `sage-tdlib`) and then filter by distribution.

Answer 24 · 2020-06-03T23:32:00.000Z

Description changed:

--- 
+++ 
@@ -5,7 +5,7 @@
 We remove `OptionalExtension`s as follows. We map installed packages to "distributions" (for example, `tdlib` -> `sage-tdlib`) and then filter by distribution.
 
 Follow-up tickets:
-- as part of Meta-ticket #29705, we will make these "distributions" actually separate distutils packages
+- as part of Meta-ticket #29705, we will make these "distributions" actually separate distutils packages that provide namespace packages (#28925)
 - remove the file `module_list.py` (not done on this ticket to avoid possible merge conflicts), deprecate `OptionalExtension` (which may be in use by user packages?)

Answer 25 · 2020-06-04T00:58:26.000Z

Changed commit from ec7e9c5 to 25a2340

Answer 26 · 2020-06-04T00:58:26.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`38b6bcf`	`Merge tag '9.2.beta0' into t/29411/make_sagelib_a_script_package`
`f9a30f6`	`build/pkgs/sagelib/spkg-install: Fix up error exits`
`00a1d57`	`Merge branch 't/29411/make_sagelib_a_script_package' into t/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup`
`25a2340`	`Merge branch 't/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' into t/29701/replace_use_of_module_list_optionalextension`

Answer 27 · 2020-06-04T02:26:57.000Z

Changed commit from 25a2340 to 2d866a3

Answer 28 · 2020-06-04T02:26:57.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`2d866a3`	`src/sage/graphs/graph_decompositions/tdlib.pyx: Fixup`

Answer 29 · 2020-06-07T18:48:57.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`b1b3787`	`sage.env.cython_aliases: Fix for systems without zlib pc`
`802356a`	`Merge branch 't/29706/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files' into t/29791/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_6__last_`
`3beb5b7`	`Merge branch 't/29791/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_6__last_' into t/29701/replace_use_of_module_list_optionalextension`

Answer 30 · 2020-06-07T18:48:57.000Z

Changed commit from 2d866a3 to 3beb5b7

Answer 31 · 2020-06-07T18:53:54.000Z

Changed commit from 3beb5b7 to 7f8850a

Answer 32 · 2020-06-07T18:53:54.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`7f8850a`	`src/module_list.py: Document that it is obsolete`

Answer 33 · 2020-06-07T20:30:43.000Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`5db5318`	`Trac #29345: remove "break" statements from AC_SEARCH_LIBS.`
`e810ad1`	`Trac #29345: don't use sage's config.status for the lrcalc build.`
`93c9921`	`Trac #29345: replace the function that populates the CVXOPT_* variables.`
`0e66a0a`	`Trac #29345: add Dima's SPKG patches for ksh compatibility.`
`df3f05e`	`build/make/Makefile.in [SCRIPT_PACKAGE_templ]: cd into the SPKG directory; adjust spkg-install scripts`
`5372065`	`Merge branch 't/29793/script_packages_should_cd_into_the_spkg_directory' into t/29411/make_sagelib_a_script_package`
`c166b97`	`Merge branch 't/29411/make_sagelib_a_script_package' into t/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup`
`cc30471`	`build/bin/write-dockerfile.sh: Do not ADD removed file src/Makefile.in`
`8a41326`	`Merge branch 't/29411/make_sagelib_a_script_package' into t/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup`
`26a85a4`	`Merge branch 't/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' into t/29701/replace_use_of_module_list_optionalextension`

Answer 34 · 2020-06-07T20:30:43.000Z

Changed commit from 7f8850a to 26a85a4

Answer 35 · 2020-06-08T02:22:04.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`5d5803e`	`src/sage/graphs/planarity.pyx: Add forgotten distutils directive`
`a0be9b6`	`Merge branch 't/29790/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_5__sage_graphs_' into t/29701/replace_use_of_module_list_optionalextension`

Answer 36 · 2020-06-08T02:22:04.000Z

Changed commit from 26a85a4 to a0be9b6

Answer 37 · 2020-06-08T02:39:11.000Z

comment:25

Tests run at https://github.com/mkoeppe/sage/actions/runs/128112212

Answer 38 · 2020-06-26T16:36:53.000Z

Changed commit from a0be9b6 to 14396af

Answer 39 · 2020-06-26T16:36:53.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`14396af`	`Merge tag '9.2.beta2' into t/29701/replace_use_of_module_list_optionalextension`

Answer 40 · 2020-06-26T16:38:21.000Z

Changed dependencies from #29702 (#29411), #29706, #29720, #29721, #29785, #29786, #29790, #29791 to #29702 (#29411), #29786, #29790, #29791

Answer 41 · 2020-06-26T16:38:21.000Z

comment:27

Merged in latest beta. Needs review

Answer 42 · 2020-06-28T22:05:46.000Z

Changed commit from 14396af to d62da15

Answer 43 · 2020-06-28T22:05:46.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`d62da15`	`sage_setup.command.sage_build: Add the extensions to the distribution`

Answer 44 · 2020-06-29T09:37:27.000Z

comment:29

I'm a bit confused:

find_python_sources gives you all sources including all with optional packages if distributions=None.

However, if distributions is sage-tdlib, you get only one source file.

Don't we want find_python_sources to contain everything except for not installed optional packages?

Answer 45 · 2020-06-29T17:18:02.000Z

comment:30

Replying to @kliem:

find_python_sources gives you all sources including all with optional packages if distributions=None.

However, if distributions is sage-tdlib, you get only one source file.

That's right. This is a feature for modularization as in #29864 (Modularization of sagelib: Break out a separate package sage-tdlib).

Don't we want find_python_sources to contain everything except for not installed optional packages?

For this, one passes distributions=['', 'sage-tdlib']. The empty string designates all source files without a sage_setup: distribution directive. This list of distributions is computed in src/setup.py.

Answer 46 · 2020-06-29T18:04:28.000Z

comment:31

I see.

I would suggest updating the benchmark. Also I can confirm this, it is not really meaningful, as this is not the normal use case, isn't it. The normal use case would be with distributions=[''] and this takes about 500 ms and not 20 ms. Still no problem I think, but much longer.

Answer 47 · 2020-06-29T18:10:36.000Z

comment:32

Otherwise the source code looks fine. Do we have any test runs? Should I run them?

Answer 48 · 2020-06-29T19:09:49.000Z

comment:33

I have tested the branch of this ticket on macOS and also as part of #29950 and other tickets.

Answer 49 · 2020-06-29T19:14:49.000Z

comment:34

Replying to @kliem:

I would suggest updating the benchmark.

Good idea, will do

Answer 50 · 2020-06-29T19:20:02.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`06a3609`	`sage_setup.find.find_python_sources: Add benchmark doctest`

Answer 51 · 2020-06-29T19:20:02.000Z

Changed commit from d62da15 to 06a3609

Answer 52 · 2020-06-29T19:40:30.000Z

comment:36

As I have capacities I started test for only this ticket:

https://github.com/kliem/sage/pull/18/checks

Answer 53 · 2020-07-01T15:34:39.000Z

comment:37

There is that annoying failure of sageinspect again

ubuntu-bionic, minimal (for example, but looks like it is anywhere)

File "src/sage/misc/sageinspect.py", line 28, in sage.misc.sageinspect
Failed example:
    sage_getsource(sage.rings.rational)[5:]
Expected:
    'Rational Numbers...'
Got:
    'tutils: libraries = ntl\nr"""\nRational Numbers\n\nAUTHORS:\n\n- William Stein (2005): first version\n\n- William Stein (2006-02-22): floor and ceil (pure fast GMP versions).\n\n- Gonzalo Tornaria and William Stein (2006-03-02): greatly improved\n  python/GMP conversion; hashing\n\n- William Stein and Naqi Jaffery (2006-03-06): height, sqrt examples,\n  and improve behavior of sqrt.\n\n- David Harvey (2006-09-15): added nth_root\n\n- Pablo De Napoli (2007-04-01): corrected the implementations of\n  multiplicative_order, is_one; optimized __nonzero__ ; documented:\n  lcm,gcd\n\n- John Cremona (2009-05-15): added support for local and global\n  logarithmic heights.\n\n- Travis Scrimshaw (2012-10-18): Added doctests for full coverage.\n\n- Vincent Delecroix (2013): continued fraction\n\n- Vincent Delecroix (2017-05-03): faster integer-rational comparison\n\n- Vincent Klein (2017-05-11): add __mpq__() to class Rational\n\n- Vincent Klein (2017-05-22): Rational constructor support gmpy2.mpq\n  or gmpy2.mpz parameter. Add __mpz__ to class Rational.\n\nTESTS::\n\n    sage: a = -2/3\n    sage: a == loads(dumps(a))\n    True\n"""\n\n# ****************************************************************************\n#       Copyright (C) 2004, 2006 William Stein <wstein@gmail.com>\n#       Copyright (C) 2017 Vincent Delecroix <20100.delecroix@gmail.com>\n#\n#  Distributed under the terms of the GNU General Public License (GPL)\n#  as published by the Free Software Foundation; either version 2 of\n#  the License, or (at your option) any later version.\n#                  https://www.gnu.org/licenses/\n# ****************************************************************************\n\ncimport cython\nfrom cpython cimport *\nfrom cpython.object cimport Py_EQ, Py_NE\n\nfrom cysignals.signals cimport sig_on, sig_off\n\nimport operator\nimport fractions\n\nfrom sage.misc.mathml import mathml\nfrom sage.arith.long cimport pyobject_to_long, integer_check_long_py\nfrom sage.cpython.string cimport char_to_str, str_to_bytes\n\nimport sage.misc.misc as misc\nfrom sage.structure.sage_object cimport SageObject\nfrom sage.structure.richcmp cimport rich_to_bool_sgn\nimport sage.rings.rational_field\n\ncimport sage.rings.integer as integer\nfrom .integer cimport Integer\n\nfrom cypari2.paridecl cimport *\nfrom cypari2.gen cimport Gen as pari_gen\nfrom sage.libs.pari.convert_gmp cimport INT_to_mpz, INTFRAC_to_mpq, new_gen_from_mpq_t\n\nfrom .integer_ring import ZZ\nfrom sage.arith.rational_reconstruction cimport mpq_rational_reconstruction\n\nfrom sage.structure.coerce cimport is_numpy_type\n\nfrom sage.libs.gmp.pylong cimport mpz_set_pylong\n\nfrom sage.structure.coerce cimport coercion_model\nfrom sage.structure.element cimport Element\nfrom sage.structure.element import coerce_binop\nfrom sage.structure.parent cimport Parent\nfrom sage.categories.morphism cimport Morphism\nfrom sage.categories.map cimport Map\n\n\n\nimport sage.rings.real_mpfr\nimport sage.rings.real_double\nfrom libc.stdint cimport uint64_t\nfrom sage.libs.gmp.binop cimport mpq_add_z, mpq_mul_z, mpq_div_zz\n\nfrom cpython.int cimport PyInt_AS_LONG\n\ncimport sage.rings.fast_arith\nimport  sage.rings.fast_arith\n\ncdef sage.rings.fast_arith.arith_int ai\nai = sage.rings.fast_arith.arith_int()\n\ncdef object numpy_long_interface = {\'typestr\': \'=i4\' if sizeof(long) == 4 else \'=i8\' }\ncdef object numpy_int64_interface = {\'typestr\': \'=i8\'}\ncdef object numpy_object_interface = {\'typestr\': \'|O\'}\ncdef object numpy_double_interface = {\'typestr\': \'=f8\'}\n\nfrom libc.math cimport ldexp\nfrom sage.libs.gmp.all cimport *\n\ncimport gmpy2\ngmpy2.import_gmpy2()\n\n\ncdef class Rational(sage.structure.element.FieldElement)\n'

Answer 54 · 2020-07-02T03:58:59.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`43a9b16`	`src/sage/misc/sageinspect.py: Fix up doctest that depends on a modified source file`
`a5bc828`	`src/sage/misc/sageinspect.py: Fixup fixup`
`1baaa68`	`src/sage/misc/sageinspect.py: Remove unused import`
`174626c`	`Merge branch 't/29786/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_4__sage_rings_' into t/29701/replace_use_of_module_list_optionalextension`
`a56dc35`	`Merge tag '9.2.beta1' into t/29702/public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup`
`034a7f7`	`Merge branch 'public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' of git://trac.sagemath.org/sage into t/29701/replace_use_of_module_list_optionalextension`

Answer 55 · 2020-07-02T03:58:59.000Z

Changed commit from 06a3609 to 034a7f7

Answer 56 · 2020-07-02T04:00:06.000Z

comment:39

Thanks for catching this. I have merged in the latest branches of the dependencies. The updated #29786 where this error came from should fix this.

Answer 57 · 2020-07-02T05:52:41.000Z

comment:40

I was wondering, where this error came from now. Good to know that it is entirely harmless.

Replying to @mkoeppe:

Thanks for catching this. I have merged in the latest branches of the dependencies. The updated #29786 where this error came from should fix this.

Answer 58 · 2020-07-02T05:53:11.000Z

comment:41

LGTM.

Answer 59 · 2020-07-02T05:53:34.000Z

Reviewer: Jonathan Kliem

Answer 60 · 2020-07-02T19:53:38.000Z

comment:43

Thank you!

Answer 61 · 2020-07-04T16:18:04.000Z

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

`d3c608b`	`Merge tag '9.2.beta3' into t/29701/replace_use_of_module_list_optionalextension`

Answer 62 · 2020-07-04T16:18:04.000Z

Changed commit from 034a7f7 to d3c608b

Answer 63 · 2020-07-04T16:18:30.000Z

comment:45

Merged 9.2.beta3

Answer 64 · 2020-07-26T22:51:24.000Z

comment:47

Volker, is there something missing on this ticket that keeps it from being merged?

Answer 65 · 2020-08-02T18:05:29.000Z

Changed dependencies from #29702 (#29411), #29786, #29790, #29791 to #29702, #29786, #29790, #29791

Answer 66 · 2020-08-02T18:05:29.000Z

comment:49

#29411 isn't merged (the release script can only check whether a ticket has been merged if you think about it; Do not depend on invalid/duplicate/meta-tickets, or anything else that doesn't actually contribute commits). Also (#29411) isn't supported as dependency, whatever the parens are supposed to mean.

Answer 67 · 2020-08-02T18:13:11.000Z

comment:50

#29411 is a real ticket with 9.2 as milestone, marked as closed, with a branch which looks like a merged branch. Is it the same kind of issue as with #21226 ?

Answer 68 · 2020-08-02T18:45:12.000Z

comment:51

Thanks for cleaning up the dependency list.

Unfortunately the format does not seem to be documented anywhere.

(#29411 is a regular merged ticket. You merged it in 0bcd001)

Answer 69 · 2020-08-02T22:22:55.000Z

comment:52

Yes, in this case it was the parens.

Answer 70 · 2020-08-02T22:58:15.000Z

comment:53

sage -t --long --warn-long 40.4 --random-seed=0 src/sage_setup/clean.py
**********************************************************************
File "src/sage_setup/clean.py", line 83, in sage_setup.clean._find_stale_files
Failed example:
    python_packages, python_modules = find_python_sources(
        SAGE_SRC, ['sage', 'sage_setup'])
Exception raised:
    Traceback (most recent call last):
      File "/home/release/Sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 715, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/release/Sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1139, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage_setup.clean._find_stale_files[3]>", line 2, in <module>
        SAGE_SRC, ['sage', 'sage_setup'])
    ValueError: too many values to unpack (expected 2)
**********************************************************************
File "src/sage_setup/clean.py", line 92, in sage_setup.clean._find_stale_files
Failed example:
    stale_iter = _find_stale_files(SAGE_LIB, python_packages, python_modules, [], extra_files)
Exception raised:
    Traceback (most recent call last):
      File "/home/release/Sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 715, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/release/Sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1139, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage_setup.clean._find_stale_files[6]>", line 1, in <module>
        stale_iter = _find_stale_files(SAGE_LIB, python_packages, python_modules, [], extra_files)
    NameError: name 'python_packages' is not defined
**********************************************************************
File "src/sage_setup/clean.py", line 95, in sage_setup.clean._find_stale_files
Failed example:
    for f in stale_iter:
        if f.endswith(skip_extensions): continue
        if '/ext_data/' in f: continue
        print('Found stale file: ' + f)
Exception raised:
    Traceback (most recent call last):
      File "/home/release/Sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 715, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/release/Sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1139, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage_setup.clean._find_stale_files[9]>", line 1, in <module>
        for f in stale_iter:
    NameError: name 'stale_iter' is not defined
**********************************************************************
1 item had failures:
   3 of  11 in sage_setup.clean._find_stale_files
    [17 tests, 3 failures, 0.03 s]
----------------------------------------------------------------------
sage -t --long --warn-long 40.4 --random-seed=0 src/sage_setup/clean.py  # 3 doctests failed
----------------------------------------------------------------------

Answer 71 · 2020-08-03T00:26:53.000Z

Changed commit from d3c608b to 55c3fbc

Answer 72 · 2020-08-03T00:26:53.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`fcad518`	`Merge tag '9.2.beta7' into t/29701/replace_use_of_module_list_optionalextension`
`55c3fbc`	`src/sage_setup/clean.py: Fix doctest`

Answer 73 · 2020-08-03T00:40:30.000Z

Changed dependencies from #29702, #29786, #29790, #29791 to none

Answer 74 · 2020-08-03T05:38:31.000Z

comment:57

Back to positive.

Answer 75 · 2020-08-03T17:43:26.000Z

comment:58

Thanks!

Answer 76 · 2020-08-04T22:17:07.000Z

comment:59

[dochtml] Build finished. The built documents can be found in /home/release/Sage/local/share/doc/sage/inventory/en/reference/manifolds
[dochtml] [algebras ] updating environment: 0 added, 5 changed, 0 removed
[dochtml] Warning: Could not import sage.graphs.graph_generators cannot import name 'distance_regular' from 'sage.graphs' (/home/release/Sage/local/lib/python3.7/site-packages/sage/graphs/__init__.py)
[dochtml] [arithgrou] The inventory files are in local/share/doc/sage/inventory/en/reference/arithgroup.
[dochtml] Build finished. The built documents can be found in /home/release/Sage/local/share/doc/sage/inventory/en/reference/arithgroup
[dochtml] Error building the documentation.
[dochtml] Traceback (most recent call last):
[dochtml]   File "/home/release/Sage/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
[dochtml]     "__main__", mod_spec)
[dochtml]   File "/home/release/Sage/local/lib/python3.7/runpy.py", line 85, in _run_code
[dochtml]     exec(code, run_globals)
[dochtml]   File "/home/release/Sage/local/lib/python3.7/site-packages/sage_setup/docbuild/__main__.py", line 2, in <module>
[dochtml]     main()
[dochtml]   File "/home/release/Sage/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 1721, in main
[dochtml]     builder()
[dochtml]   File "/home/release/Sage/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 328, in _wrapper
[dochtml]     getattr(get_builder(document), 'inventory')(*args, **kwds)
[dochtml]   File "/home/release/Sage/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 553, in _wrapper
[dochtml]     self._build_everything_except_bibliography(lang, format, *args, **kwds)
[dochtml]   File "/home/release/Sage/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 539, in _build_everything_except_bibliography
[dochtml]     build_many(build_ref_doc, non_references)
[dochtml]   File "/home/release/Sage/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 281, in build_many
[dochtml]     _build_many(target, args, processes=NUM_THREADS)
[dochtml]   File "/home/release/Sage/local/lib/python3.7/site-packages/sage_setup/docbuild/utils.py", line 283, in build_many
[dochtml]     raise worker_exc.original_exception
[dochtml] ImportError: cannot import name 'distance_regular' from 'sage.graphs' (/home/release/Sage/local/lib/python3.7/site-packages/sage/graphs/__init__.py)
[dochtml] 
[dochtml]     Note: incremental documentation builds sometimes cause spurious
[dochtml]     error messages. To be certain that these are real errors, run
[dochtml]     "make doc-clean" first and try again.
make[3]: *** [Makefile:1816: doc-html] Error 1
make[3]: Leaving directory '/home/release/Sage/build/make'
make[2]: *** [Makefile:1707: all-start] Error 2
make[2]: Leaving directory '/home/release/Sage/build/make'

Answer 77 · 2020-08-04T22:20:28.000Z

comment:60

Replying to @vbraun:

[dochtml] Build finished. The built documents can be found in /home/release/Sage/local/share/doc/sage/inventory/en/reference/manifolds
[dochtml] [algebras ] updating environment: 0 added, 5 changed, 0 removed
[dochtml] Warning: Could not import sage.graphs.graph_generators cannot import name 'distance_regular' from 'sage.graphs' (/home/release/Sage/local/lib/python3.7/site-packages/sage/graphs/__init__.py)
[dochtml] [arithgrou] The inventory files are in local/share/doc/sage/inventory/en/reference/arithgroup.
[dochtml] Build finished. The built documents can be found in /home/release/Sage/local/share/doc/sage/inventory/en/reference/arithgroup
[dochtml] Error building the documentation.
[dochtml] Traceback (most recent call last):
[dochtml]   File "/home/release/Sage/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
[dochtml]     "__main__", mod_spec)
[dochtml]   File "/home/release/Sage/local/lib/python3.7/runpy.py", line 85, in _run_code
[dochtml]     exec(code, run_globals)
[dochtml]   File "/home/release/Sage/local/lib/python3.7/site-packages/sage_setup/docbuild/__main__.py", line 2, in <module>
[dochtml]     main()
[dochtml]   File "/home/release/Sage/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 1721, in main
[dochtml]     builder()
[dochtml]   File "/home/release/Sage/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 328, in _wrapper
[dochtml]     getattr(get_builder(document), 'inventory')(*args, **kwds)
[dochtml]   File "/home/release/Sage/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 553, in _wrapper
[dochtml]     self._build_everything_except_bibliography(lang, format, *args, **kwds)
[dochtml]   File "/home/release/Sage/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 539, in _build_everything_except_bibliography
[dochtml]     build_many(build_ref_doc, non_references)
[dochtml]   File "/home/release/Sage/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 281, in build_many
[dochtml]     _build_many(target, args, processes=NUM_THREADS)
[dochtml]   File "/home/release/Sage/local/lib/python3.7/site-packages/sage_setup/docbuild/utils.py", line 283, in build_many
[dochtml]     raise worker_exc.original_exception
[dochtml] ImportError: cannot import name 'distance_regular' from 'sage.graphs' (/home/release/Sage/local/lib/python3.7/site-packages/sage/graphs/__init__.py)
[dochtml] 
[dochtml]     Note: incremental documentation builds sometimes cause spurious
[dochtml]     error messages. To be certain that these are real errors, run
[dochtml]     "make doc-clean" first and try again.
make[3]: *** [Makefile:1816: doc-html] Error 1
make[3]: Leaving directory '/home/release/Sage/build/make'
make[2]: *** [Makefile:1707: all-start] Error 2
make[2]: Leaving directory '/home/release/Sage/build/make'

Are you sure? I thought it would have come from either #30240 or #30260 since they both touch "distance_regular".

Answer 78 · 2020-08-04T22:37:43.000Z

comment:61

#30260 is broken. It is confused about whether it wants sage.graphs.distance_regular or sage.graphs.generators.distance_regular

Answer 79 · 2020-08-04T22:56:31.000Z

comment:63

Replying to @mkoeppe:

#30260 is broken. It is confused about whether it wants sage.graphs.distance_regular or sage.graphs.generators.distance_regular

it used to be broken, but IMHO it's fixed by #30240 comment:32

Answer 80 · 2020-08-04T22:58:16.000Z

comment:64

Replying to @dimpase:

Replying to @mkoeppe:

#30260 is broken. It is confused about whether it wants sage.graphs.distance_regular or sage.graphs.generators.distance_regular

it used to be broken, but IMHO it's fixed by #30240 comment:32

Unfortunately Volker merged before that.

Answer 81 · 2020-08-04T23:00:49.000Z

comment:65

#30240 does not even have #30260 as a dependency...

Answer 82 · 2020-08-05T09:03:23.000Z

comment:66

Replying to @mkoeppe:

#30240 does not even have #30260 as a dependency...

Should it? #30260 is a later ticket and it depends on #30240, but #30240 doesn't rely on #30260.
I apologise for the confusion but when #30240 went from "positive review" to "needs work" and I forgot to update #30260.

Answer 83 · 2020-08-05T14:30:16.000Z

comment:67

Replying to @Ivo-Maffei:

Replying to @mkoeppe:

#30240 does not even have #30260 as a dependency...

Should it? #30260 is a later ticket and it depends on #30240, but #30240 doesn't rely on #30260.
I apologise for the confusion but when #30240 went from "positive review" to "needs work" and I forgot to update #30260.

That's alright -- thanks for the fast reaction and fix!

Answer 84 · 2020-08-08T09:16:50.000Z

comment:68

sage -t --long --warn-long 42.4 --random-seed=0 src/sage_setup/find.py
**********************************************************************
File "src/sage_setup/find.py", line 215, in sage_setup.find.find_extra_files
Failed example:
    extras["sage/ext/interpreters"]
Expected:
    ['.../src/sage/ext/interpreters/wrapper_cdf.pxd', ...wrapper_cdf.h...]
Got:
    ['/home/release/Sage/src/sage/ext/interpreters/wrapper_cc.pxd',
     '/home/release/Sage/src/sage/ext/interpreters/wrapper_cc.pyx',
     '/home/release/Sage/src/sage/ext/interpreters/wrapper_cdf.pxd',
     '/home/release/Sage/src/sage/ext/interpreters/wrapper_cdf.pyx',
     '/home/release/Sage/src/sage/ext/interpreters/wrapper_el.pxd',
     '/home/release/Sage/src/sage/ext/interpreters/wrapper_el.pyx',
     '/home/release/Sage/src/sage/ext/interpreters/wrapper_py.pxd',
     '/home/release/Sage/src/sage/ext/interpreters/wrapper_py.pyx',
     '/home/release/Sage/src/sage/ext/interpreters/wrapper_rdf.pxd',
     '/home/release/Sage/src/sage/ext/interpreters/wrapper_rdf.pyx',
     '/home/release/Sage/src/sage/ext/interpreters/wrapper_rr.pxd',
     '/home/release/Sage/src/sage/ext/interpreters/wrapper_rr.pyx']
**********************************************************************
1 item had failures:
   1 of   7 in sage_setup.find.find_extra_files
    [36 tests, 1 failure, 0.27 s]
----------------------------------------------------------------------
sage -t --long --warn-long 42.4 --random-seed=0 src/sage_setup/find.py  # 1 doctest failed
----------------------------------------------------------------------

Answer 85 · 2020-08-08T12:23:12.000Z

comment:69

Might have been from #29950

Answer 86 · 2020-08-08T20:41:37.000Z

comment:70

Yes, it's #29950 that broke it.

Answer 87 · 2020-08-10T18:50:17.000Z

Changed branch from u/mkoeppe/replace_use_of_module_list_optionalextension to 55c3fbc