Meta-ticket: Eliminate use of patches for Python packages

[mkoeppe]

Installing "sdist + patches" is not supported by Python package infrastructure.

We review existing patches, and for those packages that still need them, we create a repository fork and create and deposit an sdist tarball from here.

Tickets:

• #31967/#31280 rst2ipynb: Update to 0.2.3 to remove patching
• #32691 setuptools: Upgrade to 58.2.0, remove old easy_install patch
• #32832 cython: Eliminate use of custom patches
• #32828 pyzmq

Packages with patches can be found as follows:

$ ./sage -package list --has-file install-requires.txt --has-file patches | sort
cvxopt
cypari
cython
dot2tex
ipywidgets
notedown
numpy
pillow
psutil
rpy2
setuptools
sphinx
widgetsnbextension

Also, some Python packages need specific settings of environment variables at build time. Example: #21170 (pypolymake); a new package script such as spkg-env could set these (#29045)

Finally, some packages may install custom configurations, such as the notebook package's jupyter_notebook_config.py that currently gets installed into "$SAGE_INST_LOCAL"/etc/jupyter.

CC: @dimpase @orlitzky @tobiasdiez

Component: build

Issue created by migration from https://trac.sagemath.org/ticket/31543

[mkoeppe]

Description changed:

--- 
+++ 
@@ -20,4 +20,7 @@
 
 We review existing patches, and for those packages that still need them, we create a repository fork and create and deposit an sdist tarball from here.
 
+Also, some Python packages need specific settings of environment variables at build time. Example: #21170 (pypolymake); a new package script such as `spkg-env` could set these (#29045)
 
+
+

[fchapoton]

comment:2

rst2ipynb could be updated to 0.2.3, that contains our patch

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,4 +1,12 @@
 Installing "sdist + patches" is not supported by Python package infrastructure.
+
+We review existing patches, and for those packages that still need them, we create a repository fork and create and deposit an sdist tarball from here.
+
+Tickets:
+
+- #31967 rst2ipynb: Update to 0.2.3 to remove patching
+
+Packages with patches can be found as follows:
 
 ```
 $ ./sage -package list --has-file install-requires.txt --has-file patches
@@ -18,8 +26,6 @@
 psutil
 ```
 
-We review existing patches, and for those packages that still need them, we create a repository fork and create and deposit an sdist tarball from here.
-
 Also, some Python packages need specific settings of environment variables at build time. Example: #21170 (pypolymake); a new package script such as `spkg-env` could set these (#29045)
 
 

[mkoeppe]

Description changed:

--- 
+++ 
@@ -9,21 +9,20 @@
 Packages with patches can be found as follows:
 
 ```
-$ ./sage -package list --has-file install-requires.txt --has-file patches
+$ ./sage -package list --has-file install-requires.txt --has-file patches | sort
+cvxopt
+cypari
+cython
+dot2tex
+ipywidgets
+notedown
+numpy
+pillow
+psutil
 rpy2
+setuptools
+sphinx
 widgetsnbextension
-cvxopt
-ipywidgets
-dot2tex
-cython
-numpy
-cypari
-pillow
-rst2ipynb
-sphinx
-notedown
-setuptools
-psutil
 ```
 
 Also, some Python packages need specific settings of environment variables at build time. Example: #21170 (pypolymake); a new package script such as `spkg-env` could set these (#29045)

[mkoeppe]

Description changed:

--- 
+++ 
@@ -4,7 +4,7 @@
 
 Tickets:
 
-- #31967 rst2ipynb: Update to 0.2.3 to remove patching
+- #31967/#31280 rst2ipynb: Update to 0.2.3 to remove patching
 
 Packages with patches can be found as follows:
 

[mkoeppe]

Description changed:

--- 
+++ 
@@ -4,7 +4,8 @@
 
 Tickets:
 
-- #31967/#31280 rst2ipynb: Update to 0.2.3 to remove patching
+- #31967/#31280 `rst2ipynb`: Update to 0.2.3 to remove patching
+- #32691 `setuptools`: Upgrade to 58.2.0, remove old `easy_install` patch
 
 Packages with patches can be found as follows:
 

[orlitzky]

comment:10

The pillow patch might not be needed any longer, but someone with macOS will have to check.

[tobiasdiez]

comment:11

In #30371, no patches to python packages are applied (as the packages used there come directly from pypi completely bypassing sages package infrastructure).
The only real test failures on ubuntu are:

File "sage/matrix/matrix_integer_dense.pyx", line 3706, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.determinant
Failed example:
    A.determinant(algorithm='linbox',proof=False) == d
Expected:
    True
Got:
    False
**********************************************************************
File "sage/cpython/dict_del_by_value.pyx", line 268, in sage.cpython.dict_del_by_value.test_del_dictitem_by_exact_value
Failed example:
    test_del_dictitem_by_exact_value(D, ZZ, 2)
Exception raised:
    Traceback (most recent call last):
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.cpython.dict_del_by_value.test_del_dictitem_by_exact_value[6]>", line 1, in <module>
        test_del_dictitem_by_exact_value(D, ZZ, Integer(2))
      File "sage/cpython/dict_del_by_value.pyx", line 275, in sage.cpython.dict_del_by_value.test_del_dictitem_by_exact_value
        del_dictitem_by_exact_value(<PyDictObject *>D, <PyObject *>value, h)
    TypeError: an integer is required
**********************************************************************
File "sage/cpython/dict_del_by_value.pyx", line 271, in sage.cpython.dict_del_by_value.test_del_dictitem_by_exact_value
Failed example:
    test_del_dictitem_by_exact_value(D, QQ, 1)
Exception raised:
    Traceback (most recent call last):
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.cpython.dict_del_by_value.test_del_dictitem_by_exact_value[8]>", line 1, in <module>
        test_del_dictitem_by_exact_value(D, QQ, Integer(1))
      File "sage/cpython/dict_del_by_value.pyx", line 275, in sage.cpython.dict_del_by_value.test_del_dictitem_by_exact_value
        del_dictitem_by_exact_value(<PyDictObject *>D, <PyObject *>value, h)
    TypeError: an integer is required

That's good evidence that most patches might no longer be needed (at least on ubuntu and possibly updating the python packages).

[mkoeppe]

Description changed:

--- 
+++ 
@@ -6,6 +6,7 @@
 
 - #31967/#31280 `rst2ipynb`: Update to 0.2.3 to remove patching
 - #32691 `setuptools`: Upgrade to 58.2.0, remove old `easy_install` patch
+- #32832 `cython`: Eliminate use of custom patches
 
 Packages with patches can be found as follows:
 

[mkoeppe]

Description changed:

--- 
+++ 
@@ -7,6 +7,7 @@
 - #31967/#31280 `rst2ipynb`: Update to 0.2.3 to remove patching
 - #32691 `setuptools`: Upgrade to 58.2.0, remove old `easy_install` patch
 - #32832 `cython`: Eliminate use of custom patches
+- #32828 `pyzmq`
 
 Packages with patches can be found as follows:
 

[orlitzky]

Description changed:

--- 
+++ 
@@ -30,5 +30,5 @@
 
 Also, some Python packages need specific settings of environment variables at build time. Example: #21170 (pypolymake); a new package script such as `spkg-env` could set these (#29045)
 
+Finally, some packages may install custom configurations, such as the `notebook` package's `jupyter_notebook_config.py` that currently gets installed into ` "$SAGE_INST_LOCAL"/etc/jupyter`.
 
-

[orlitzky]

I think we are all using unpatched python deps these days.