Meta-ticket: Clear out init.py files in preparation for namespace packages

Question

Meta-ticket: Clear out init.py files in preparation for namespace packages

mkoeppe opened this issue 4 years ago · 21 comments

(Part of the modularization effort, #29705)

Whenever there are two distribution packages that provide modules with a common prefix of Python packages, that prefix needs to be a native namespace package, i.e., there cannot be an __init__.py file.

For example,

sagemath-tdlib will provide sage.graphs.graph_decompositions.tdlib,
sagemath-rw will provide sage.graphs.graph_decompositions.rankwidth,
sagemath-graphs will provide all of the rest of sage.graphs.graph_decompositions (and most of sage.graphs).
Then, none of
sage,
sage.graphs,
sage.graphs.graph_decomposition
can be an ordinary Python package (with __init__.py file), but rather each of them has to be a native namespace package (no __init__.py file).

In this meta-ticket, we make sure that all such __init__.py files are empty. (The actual removal of empty __init__.py files will be done in #28925.)

Not all __init__.py files need to be empty. For example,

all of sage.cpython and sage.structure (after #29870) is intended to be in one distribution package, sagemath-objects. Hence sage.cpython and sage.structure keep their nonempty __init__.py files, and we may even add a comment that documents that they are intended to stay ordinary packages. (#31420 clears sage.__init__ and adds such a comment to sage.cpython.__init__)
sage.libs.giac is intended to be in one distribution package, sagemath-giac. Hence sage.libs.giac will keep its nonempty __init__.py file (but sage.libs will not).

With positively-reviewed tickets #31420/#32479/#32506/#32507

$ find src/sage -name '__init__.py' | xargs wc -l | grep -v '^ *0'
       2 src/sage/crypto/__init__.py
       3 src/sage/crypto/mq/__init__.py
       3 src/sage/dynamics/__init__.py
       3 src/sage/dynamics/cellular_automata/__init__.py
      42 src/sage/combinat/words/__init__.py
      38 src/sage/combinat/designs/__init__.py            #32508
      13 src/sage/combinat/chas/__init__.py               #32508
      11 src/sage/combinat/path_tableaux/__init__.py      #32508
      49 src/sage/combinat/__init__.py                    #32508
      10 src/sage/combinat/cluster_algebra_quiver/__init__.py  #32508
      51 src/sage/combinat/species/__init__.py            #32508
      31 src/sage/combinat/sf/__init__.py                 #32508
      29 src/sage/combinat/crystals/__init__.py           #32508
      10 src/sage/combinat/ncsym/__init__.py              #32508
      35 src/sage/combinat/posets/__init__.py             #32508
      11 src/sage/combinat/ncsf_qsym/__init__.py          #32508
       7 src/sage/combinat/integer_lists/__init__.py            # OK, 1 distro
      36 src/sage/combinat/rigged_configurations/__init__.py  #32508
     127 src/sage/combinat/root_system/__init__.py        #32508, #32840
       9 src/sage/combinat/matrices/__init__.py
       3 src/sage/doctest/__init__.py                           # OK, 1 distro sagemath-repl
     826 src/sage/features/__init__.py                          # OK, 1 distro
       4 src/sage/repl/rich_output/__init__.py                  # OK, 1 distro sagemath-repl
      39 src/sage/__init__.py                             #32489
     359 src/sage/libs/giac/__init__.py                         # OK, 1 distro sagemath-giac
       2 src/sage/libs/ntl/__init__.py                          # OK, 1 distro
       1 src/sage/libs/gap/__init__.py                          # OK, 1 distro
     219 src/sage/libs/pari/__init__.py                         # OK, 1 distro
      55 src/sage/cpython/__init__.py                           # OK, 1 distro sagemath-objects
       2 src/sage/sat/converters/__init__.py
       4 src/sage/sat/solvers/__init__.py
       2 src/sage/matroids/__init__.py
     114 src/sage/rings/polynomial/pbori/__init__.py            # OK, 1 distro sagemath-brial
       2 src/sage/rings/polynomial/__init__.py            #32509
      10 src/sage/modules/with_basis/__init__.py                # OK, 1 distro sagemath-categories
       2 src/sage/structure/__init__.py                         # OK, 1 distro sagemath-objects
      29 src/sage/categories/__init__.py                  #32508
       2 src/sage/media/__init__.py                             # deprecated

Tickets:

#31420/#32479/#32489 sage
#32506 Remove useless 'file is not empty' comments from __init__.py files
#32507 Remove useless 'from . import all' and similar from some __init__.py files
#32508 Eliminate use of __init__.py files for supplying package docstrings
#32509 Clear out __init__.py in sage.rings.polynomial
#32840 Fix unpickle_global when register_unpickle_override appears in a module not loaded at startup

#30666 Add "sage_setup: distribution" directives to all Cython modules needing external libraries -- related because the dependencies on C/C++ libraries guide the layout of the namespace packages.

CC: @tscrim @pjbruin @kiwifb @kliem @dcoudert

Component: refactoring

Reviewer: Michael Orlitzky

Issue created by migration from https://trac.sagemath.org/ticket/32501

Answer 1 · 2021-09-10T23:21:40.000Z

Description changed:

--- 
+++ 
@@ -58,3 +58,7 @@
 Tickets:
 - #31420/#32479/#32489 `sage`
 
+Related:
+- #30666 Add "sage_setup: distribution" directives to all Cython modules needing external libraries -- related because the dependencies on C/C++ libraries guide the layout of the namespace packages.
+
+

Answer 2 · 2021-09-10T23:35:35.000Z

Description changed:

--- 
+++ 
@@ -15,7 +15,7 @@
 In this meta-ticket, we make sure that all such `__init__.py` files are empty. (The actual removal of empty `__init__.py` files will be done in #28925.)
 
 Not all `__init__.py` files need to be empty. For example, 
-- all of `sage.cpython` and `sage.structure` is intended to be in one distribution package, `sagemath-objects`. Hence `sage.cpython` and `sage.structure` keep their nonempty `__init__.py` files, and we may even add a comment that documents that they are intended to stay ordinary packages. (#31420 clears `sage.__init__` and adds such a comment to `sage.cpython.__init__`
+- all of `sage.cpython` and `sage.structure` (after #29870) is intended to be in one distribution package, `sagemath-objects`. Hence `sage.cpython` and `sage.structure` keep their nonempty `__init__.py` files, and we may even add a comment that documents that they are intended to stay ordinary packages. (#31420 clears `sage.__init__` and adds such a comment to `sage.cpython.__init__`)
 - `sage.libs.giac` is intended to be in one distribution package, `sagemath-giac`. Hence `sage.libs.giac` will keep its nonempty `__init__.py` file (but `sage.libs` will not).

Answer 3 · 2021-09-12T05:42:35.000Z

Description changed:

--- 
+++ 
@@ -57,6 +57,7 @@
 
 Tickets:
 - #31420/#32479/#32489 `sage`
+- #32506 Remove useless 'file is not empty' comments from `__init__.py` files
 
 Related:
 - #30666 Add "sage_setup: distribution" directives to all Cython modules needing external libraries -- related because the dependencies on C/C++ libraries guide the layout of the namespace packages.

Answer 4 · 2021-09-12T05:46:53.000Z

Description changed:

--- 
+++ 
@@ -58,6 +58,7 @@
 Tickets:
 - #31420/#32479/#32489 `sage`
 - #32506 Remove useless 'file is not empty' comments from `__init__.py` files
+- #32507 Remove useless 'from . import all' from some `__init__.py` files
 
 Related:
 - #30666 Add "sage_setup: distribution" directives to all Cython modules needing external libraries -- related because the dependencies on C/C++ libraries guide the layout of the namespace packages.

Answer 5 · 2021-09-12T06:32:29.000Z

Description changed:

--- 
+++ 
@@ -59,6 +59,7 @@
 - #31420/#32479/#32489 `sage`
 - #32506 Remove useless 'file is not empty' comments from `__init__.py` files
 - #32507 Remove useless 'from . import all' from some `__init__.py` files
+- #32508 Eliminate use of `__init__.py` files for supplying package docstrings
 
 Related:
 - #30666 Add "sage_setup: distribution" directives to all Cython modules needing external libraries -- related because the dependencies on C/C++ libraries guide the layout of the namespace packages.

Answer 6 · 2021-09-13T06:19:51.000Z

comment:6

A serious case:

$ cat src/sage/rings/polynomial/__init__.py
# Resolve a cyclic import
import sage.rings.convert.mpfi

We need to get rid of this one -- sage.rings.polynomial must become a namespace package

Answer 7 · 2021-09-13T06:45:25.000Z

Description changed:

--- 
+++ 
@@ -58,7 +58,7 @@
 Tickets:
 - #31420/#32479/#32489 `sage`
 - #32506 Remove useless 'file is not empty' comments from `__init__.py` files
-- #32507 Remove useless 'from . import all' from some `__init__.py` files
+- #32507 Remove useless 'from . import all' and similar from some `__init__.py` files
 - #32508 Eliminate use of `__init__.py` files for supplying package docstrings
 
 Related:

Answer 8 · 2021-09-13T17:52:09.000Z

Description changed:

--- 
+++ 
@@ -18,41 +18,48 @@
 - all of `sage.cpython` and `sage.structure` (after #29870) is intended to be in one distribution package, `sagemath-objects`. Hence `sage.cpython` and `sage.structure` keep their nonempty `__init__.py` files, and we may even add a comment that documents that they are intended to stay ordinary packages. (#31420 clears `sage.__init__` and adds such a comment to `sage.cpython.__init__`)
 - `sage.libs.giac` is intended to be in one distribution package, `sagemath-giac`. Hence `sage.libs.giac` will keep its nonempty `__init__.py` file (but `sage.libs` will not).
 
+With positively-reviewed tickets #31420/#32479/#32506/#32507
 
 ```
-$ find src/sage -name '__init__.py' | xargs wc -l | sort
-...
+$ find src/sage -name '__init__.py' | xargs wc -l | grep -v '^ *0'
+       2 src/sage/crypto/__init__.py
+       3 src/sage/crypto/mq/__init__.py
        3 src/sage/dynamics/__init__.py
        3 src/sage/dynamics/cellular_automata/__init__.py
-       3 src/sage/matroids/__init__.py
-       3 src/sage/modular/arithgroup/__init__.py
-       3 src/sage/modular/quatalg/__init__.py
-       4 src/sage/repl/rich_output/__init__.py
+      42 src/sage/combinat/words/__init__.py
+      38 src/sage/combinat/designs/__init__.py
+      13 src/sage/combinat/chas/__init__.py
+      11 src/sage/combinat/path_tableaux/__init__.py
+      49 src/sage/combinat/__init__.py                    #32508
+      10 src/sage/combinat/cluster_algebra_quiver/__init__.py
+      51 src/sage/combinat/species/__init__.py
+      31 src/sage/combinat/sf/__init__.py
+      29 src/sage/combinat/crystals/__init__.py
+      10 src/sage/combinat/ncsym/__init__.py
+      35 src/sage/combinat/posets/__init__.py
+      11 src/sage/combinat/ncsf_qsym/__init__.py
+       7 src/sage/combinat/integer_lists/__init__.py
+      36 src/sage/combinat/rigged_configurations/__init__.py
+     127 src/sage/combinat/root_system/__init__.py
+       9 src/sage/combinat/matrices/__init__.py
+       3 src/sage/doctest/__init__.py                           # OK, 1 distro sagemath-repl
+     826 src/sage/features/__init__.py                          # OK, 1 distro
+       4 src/sage/repl/rich_output/__init__.py                  # OK, 1 distro sagemath-repl
+      39 src/sage/__init__.py                             #32489
+     359 src/sage/libs/giac/__init__.py                         # OK, 1 distro sagemath-giac
+       2 src/sage/libs/ntl/__init__.py                          # OK, 1 distro
+       1 src/sage/libs/gap/__init__.py                          # OK, 1 distro
+     219 src/sage/libs/pari/__init__.py                         # OK, 1 distro
+      55 src/sage/cpython/__init__.py                           # OK, 1 distro sagemath-objects
+       2 src/sage/sat/converters/__init__.py
        4 src/sage/sat/solvers/__init__.py
-       5 src/sage/graphs/graph_decompositions/__init__.py
-       7 src/sage/combinat/integer_lists/__init__.py
-       9 src/sage/combinat/matrices/__init__.py
-      10 src/sage/combinat/cluster_algebra_quiver/__init__.py
-      10 src/sage/combinat/ncsym/__init__.py
-      10 src/sage/modules/with_basis/__init__.py
-      11 src/sage/combinat/ncsf_qsym/__init__.py
-      11 src/sage/combinat/path_tableaux/__init__.py
-      13 src/sage/combinat/chas/__init__.py
-      29 src/sage/categories/__init__.py
-      29 src/sage/combinat/crystals/__init__.py
-      31 src/sage/combinat/sf/__init__.py
-      35 src/sage/combinat/posets/__init__.py
-      36 src/sage/combinat/rigged_configurations/__init__.py
-      38 src/sage/combinat/designs/__init__.py
-      42 src/sage/combinat/words/__init__.py
-      49 src/sage/combinat/__init__.py
-      51 src/sage/combinat/species/__init__.py
-     107 src/sage/__init__.py                                     #31420/32479/#32489
-     114 src/sage/rings/polynomial/pbori/__init__.py
-     127 src/sage/combinat/root_system/__init__.py
-     219 src/sage/libs/pari/__init__.py
-     359 src/sage/libs/giac/__init__.py
-     826 src/sage/features/__init__.py
+       2 src/sage/matroids/__init__.py
+     114 src/sage/rings/polynomial/pbori/__init__.py            # OK, 1 distro sagemath-brial
+       2 src/sage/rings/polynomial/__init__.py            # !!!
+      10 src/sage/modules/with_basis/__init__.py                # OK, 1 distro sagemath-categories
+       2 src/sage/structure/__init__.py                         # OK, 1 distro sagemath-objects
+      29 src/sage/categories/__init__.py                  #32508
+       2 src/sage/media/__init__.py                             # deprecated
 ```
 
 Tickets:

Answer 9 · 2021-09-13T19:14:06.000Z

comment:9

Replying to @mkoeppe:

A serious case:
$ cat src/sage/rings/polynomial/__init__.py
# Resolve a cyclic import
import sage.rings.convert.mpfi
We need to get rid of this one -- sage.rings.polynomial must become a namespace package

This one was added it #25848 ("Some preparations for binding=True") for a stalled effort in #22747/#26254

Answer 10 · 2021-09-13T19:39:07.000Z

Description changed:

--- 
+++ 
@@ -55,7 +55,7 @@
        4 src/sage/sat/solvers/__init__.py
        2 src/sage/matroids/__init__.py
      114 src/sage/rings/polynomial/pbori/__init__.py            # OK, 1 distro sagemath-brial
-       2 src/sage/rings/polynomial/__init__.py            # !!!
+       2 src/sage/rings/polynomial/__init__.py            #32509
       10 src/sage/modules/with_basis/__init__.py                # OK, 1 distro sagemath-categories
        2 src/sage/structure/__init__.py                         # OK, 1 distro sagemath-objects
       29 src/sage/categories/__init__.py                  #32508
@@ -67,6 +67,7 @@
 - #32506 Remove useless 'file is not empty' comments from `__init__.py` files
 - #32507 Remove useless 'from . import all' and similar from some `__init__.py` files
 - #32508 Eliminate use of `__init__.py` files for supplying package docstrings
+- #32509 Clear out `__init__.py` in `sage.rings.polynomial`
 
 Related:
 - #30666 Add "sage_setup: distribution" directives to all Cython modules needing external libraries -- related because the dependencies on C/C++ libraries guide the layout of the namespace packages.

Answer 11 · 2021-10-20T17:41:24.000Z

comment:11

We can close this meta-ticket because it only tracks one open ticket.

Answer 12 · 2021-10-20T23:53:15.000Z

comment:12

I think it would be good to keep it open until the final ticket is done.

Answer 13 · 2021-11-09T05:08:27.000Z

Description changed:

--- 
+++ 
@@ -68,6 +68,7 @@
 - #32507 Remove useless 'from . import all' and similar from some `__init__.py` files
 - #32508 Eliminate use of `__init__.py` files for supplying package docstrings
 - #32509 Clear out `__init__.py` in `sage.rings.polynomial`
+- #32840 Fix `unpickle_global` when `register_unpickle_override` appears in a module not loaded at startup
 
 Related:
 - #30666 Add "sage_setup: distribution" directives to all Cython modules needing external libraries -- related because the dependencies on C/C++ libraries guide the layout of the namespace packages.

Answer 14 · 2021-11-09T05:33:36.000Z

Description changed:

--- 
+++ 
@@ -3,9 +3,9 @@
 Whenever there are two distribution packages that provide modules with a common prefix of Python packages, that prefix needs to be a native namespace package, i.e., there cannot be an `__init__.py` file. 
 
 For example, 
-- `sagemath-tdlib` will provide `sage.graphs.graph_decompositions.tdlib`,
-- `sagemath-rw` will provide `sage.graphs.graph_decompositions.rankwidth`,
-- `sagemath-graphs` will provide all of the rest of `sage.graphs.graph_decompositions` (and most of `sage.graphs`).
+- **sagemath-tdlib** will provide `sage.graphs.graph_decompositions.tdlib`,
+- **sagemath-rw** will provide `sage.graphs.graph_decompositions.rankwidth`,
+- **sagemath-graphs** will provide all of the rest of `sage.graphs.graph_decompositions` (and most of `sage.graphs`).
 Then, none of 
 - `sage`, 
 - `sage.graphs`, 
@@ -15,8 +15,8 @@
 In this meta-ticket, we make sure that all such `__init__.py` files are empty. (The actual removal of empty `__init__.py` files will be done in #28925.)
 
 Not all `__init__.py` files need to be empty. For example, 
-- all of `sage.cpython` and `sage.structure` (after #29870) is intended to be in one distribution package, `sagemath-objects`. Hence `sage.cpython` and `sage.structure` keep their nonempty `__init__.py` files, and we may even add a comment that documents that they are intended to stay ordinary packages. (#31420 clears `sage.__init__` and adds such a comment to `sage.cpython.__init__`)
-- `sage.libs.giac` is intended to be in one distribution package, `sagemath-giac`. Hence `sage.libs.giac` will keep its nonempty `__init__.py` file (but `sage.libs` will not).
+- all of `sage.cpython` and `sage.structure` (after #29870) is intended to be in one distribution package, **sagemath-objects**. Hence `sage.cpython` and `sage.structure` keep their nonempty `__init__.py` files, and we may even add a comment that documents that they are intended to stay ordinary packages. (#31420 clears `sage.__init__` and adds such a comment to `sage.cpython.__init__`)
+- `sage.libs.giac` is intended to be in one distribution package, **sagemath-giac**. Hence `sage.libs.giac` will keep its nonempty `__init__.py` file (but `sage.libs` will not).
 
 With positively-reviewed tickets #31420/#32479/#32506/#32507

Answer 15 · 2021-11-09T05:36:03.000Z

Description changed:

--- 
+++ 
@@ -40,7 +40,7 @@
       11 src/sage/combinat/ncsf_qsym/__init__.py
        7 src/sage/combinat/integer_lists/__init__.py
       36 src/sage/combinat/rigged_configurations/__init__.py
-     127 src/sage/combinat/root_system/__init__.py
+     127 src/sage/combinat/root_system/__init__.py        #32508, #32840
        9 src/sage/combinat/matrices/__init__.py
        3 src/sage/doctest/__init__.py                           # OK, 1 distro sagemath-repl
      826 src/sage/features/__init__.py                          # OK, 1 distro

Answer 16 · 2021-11-09T05:43:55.000Z

Description changed:

--- 
+++ 
@@ -27,19 +27,19 @@
        3 src/sage/dynamics/__init__.py
        3 src/sage/dynamics/cellular_automata/__init__.py
       42 src/sage/combinat/words/__init__.py
-      38 src/sage/combinat/designs/__init__.py
-      13 src/sage/combinat/chas/__init__.py
-      11 src/sage/combinat/path_tableaux/__init__.py
+      38 src/sage/combinat/designs/__init__.py            #32508
+      13 src/sage/combinat/chas/__init__.py               #32508
+      11 src/sage/combinat/path_tableaux/__init__.py      #32508
       49 src/sage/combinat/__init__.py                    #32508
-      10 src/sage/combinat/cluster_algebra_quiver/__init__.py
-      51 src/sage/combinat/species/__init__.py
-      31 src/sage/combinat/sf/__init__.py
-      29 src/sage/combinat/crystals/__init__.py
-      10 src/sage/combinat/ncsym/__init__.py
-      35 src/sage/combinat/posets/__init__.py
-      11 src/sage/combinat/ncsf_qsym/__init__.py
-       7 src/sage/combinat/integer_lists/__init__.py
-      36 src/sage/combinat/rigged_configurations/__init__.py
+      10 src/sage/combinat/cluster_algebra_quiver/__init__.py  #32508
+      51 src/sage/combinat/species/__init__.py            #32508
+      31 src/sage/combinat/sf/__init__.py                 #32508
+      29 src/sage/combinat/crystals/__init__.py           #32508
+      10 src/sage/combinat/ncsym/__init__.py              #32508
+      35 src/sage/combinat/posets/__init__.py             #32508
+      11 src/sage/combinat/ncsf_qsym/__init__.py          #32508
+       7 src/sage/combinat/integer_lists/__init__.py            # OK
+      36 src/sage/combinat/rigged_configurations/__init__.py  #32508
      127 src/sage/combinat/root_system/__init__.py        #32508, #32840
        9 src/sage/combinat/matrices/__init__.py
        3 src/sage/doctest/__init__.py                           # OK, 1 distro sagemath-repl

Answer 17 · 2021-11-09T05:45:31.000Z

Description changed:

--- 
+++ 
@@ -38,7 +38,7 @@
       10 src/sage/combinat/ncsym/__init__.py              #32508
       35 src/sage/combinat/posets/__init__.py             #32508
       11 src/sage/combinat/ncsf_qsym/__init__.py          #32508
-       7 src/sage/combinat/integer_lists/__init__.py            # OK
+       7 src/sage/combinat/integer_lists/__init__.py            # OK, 1 distro
       36 src/sage/combinat/rigged_configurations/__init__.py  #32508
      127 src/sage/combinat/root_system/__init__.py        #32508, #32840
        9 src/sage/combinat/matrices/__init__.py

Answer 18 · 2021-12-01T01:33:10.000Z

Author: Michael Orlitzky

Answer 19 · 2021-12-01T01:33:10.000Z

comment:18

Everything in the "Tickets:" list has been closed: fixed.

Answer 20 · 2021-12-01T01:33:22.000Z

Changed author from Michael Orlitzky to none

Answer 21 · 2021-12-01T01:33:22.000Z

Reviewer: Michael Orlitzky