Update to matplotlib 3
Closed this issue · 78 comments
Update to the latest matplotlib, currently version 3.2.1. This is Python 3-only, so it will be for Sage 9.2. This is necessary for upgrading Sphinx (#28856).
Component: packages: standard
Author: John Palmieri, François Bissey
Branch/Commit: eee7843
Reviewer: Dave Morris, Matthias Koeppe
Issue created by migration from https://trac.sagemath.org/ticket/29547
Description changed:
---
+++
@@ -1 +1 @@
-Update to the latest matplotlib, currently version 3.2.1. This is Python 3-only, so it will be for Sphinx 9.2. This is necessary for upgrading Sphinx.
+Update to the latest matplotlib, currently version 3.2.1. This is Python 3-only, so it will be for Sphinx 9.2. This is necessary for upgrading Sphinx (#28856).See also #28449 and #27866 comment:25.
Description changed:
---
+++
@@ -1 +1 @@
-Update to the latest matplotlib, currently version 3.2.1. This is Python 3-only, so it will be for Sphinx 9.2. This is necessary for upgrading Sphinx (#28856).
+Update to the latest matplotlib, currently version 3.2.1. This is Python 3-only, so it will be for Sage 9.2. This is necessary for upgrading Sphinx (#28856).Doctests known to fail in matplotlib 3.1+
sage -t --long --warn-long 88.1 /usr/lib/python3.7/site-packages/sage/combinat/root_system/root_lattice_realizations.py
**********************************************************************
File "/usr/lib/python3.7/site-packages/sage/combinat/root_system/root_lattice_realizations.py", line 3134, in sage.combinat.root_system.root_lattice_realizations.RootLatticeRealizations.ParentMethods.?
Failed example:
L.plot_crystal(C)
Expected:
Graphics object consisting of 16 graphics primitives
Got:
doctest:warning
File "/usr/lib/python-exec/python3.7/sage-runtests", line 178, in <module>
err = DC.run()
File "/usr/lib/python3.7/site-packages/sage/doctest/control.py", line 1216, in run
self.run_doctests()
File "/usr/lib/python3.7/site-packages/sage/doctest/control.py", line 917, in run_doctests
self.dispatcher.dispatch()
File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 2033, in dispatch
self.parallel_dispatch()
File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 1925, in parallel_dispatch
w.start() # This might take some time
File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 2200, in start
super(DocTestWorker, self).start()
File "/usr/lib/python3.7/multiprocessing/process.py", line 112, in start
self._popen = self._Popen(self)
File "/usr/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/usr/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
return Popen(process_obj)
File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
code = process_obj._bootstrap()
File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 2172, in run
task(self.options, self.outtmpfile, msgpipe, self.result_queue)
File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 2504, in __call__
doctests, extras = self._run(runner, options, results)
File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 2553, in _run
result = runner.run(test)
File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 897, in run
return self._run(test, compileflags, out)
File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
exec(compiled, globs)
File "<doctest sage.combinat.root_system.root_lattice_realizations.RootLatticeRealizations.ParentMethods.?[2]>", line 1, in <module>
L.plot_crystal(C)
File "/usr/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py", line 811, in displayhook
plain_text, rich_output = self._rich_output_formatter(obj, dict())
File "/usr/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py", line 625, in _rich_output_formatter
rich_output = self._call_rich_repr(obj, rich_repr_kwds)
File "/usr/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py", line 585, in _call_rich_repr
return obj._rich_repr_(self)
File "/usr/lib/python3.7/site-packages/sage/plot/graphics.py", line 1006, in _rich_repr_
self.save, kwds, file_ext, output_container)
File "/usr/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py", line 713, in graphics_from_save
save_function(filename, **kwds)
File "/usr/lib/python3.7/site-packages/sage/misc/decorators.py", line 412, in wrapper
return func(*args, **kwds)
File "/usr/lib/python3.7/site-packages/sage/plot/graphics.py", line 3357, in save
figure.tight_layout()
File "/usr/lib/python3.7/site-packages/matplotlib/figure.py", line 2476, in tight_layout
pad=pad, h_pad=h_pad, w_pad=w_pad, rect=rect)
File "/usr/lib/python3.7/site-packages/matplotlib/tight_layout.py", line 362, in get_tight_layout_figure
pad=pad, h_pad=h_pad, w_pad=w_pad)
File "/usr/lib/python3.7/site-packages/matplotlib/tight_layout.py", line 172, in auto_adjust_subplotpars
cbook._warn_external('Tight layout not applied. The left and right '
File "/usr/lib/python3.7/site-packages/matplotlib/cbook/__init__.py", line 2055, in _warn_external
warnings.warn(message, category, stacklevel)
File "/usr/lib/python3.7/warnings.py", line 110, in _showwarnmsg
msg.file, msg.line)
:
UserWarning: Tight layout not applied. The left and right margins cannot be made large enough to accommodate all axes decorations.
Graphics object consisting of 16 graphics primitives
and
sage -t --long --warn-long 88.1 /usr/lib/python3.7/site-packages/sage/plot/multigraphics.py
**********************************************************************
File "/usr/lib/python3.7/site-packages/sage/plot/multigraphics.py", line 1297, in sage.plot.multigraphics.GraphicsArray.position
Failed example:
G.position(0) # tol 1.0e-13
Expected:
(0.028906249999999998,
0.038541666666666696,
0.45390624999999996,
0.9229166666666667)
Got:
(0.023437500000000003,
0.03415488992713045,
0.4627803348994754,
0.9345951100728696)
Tolerance exceeded in 4 of 4:
0.028906249999999998 vs 0.023437500000000003, tolerance 2e-1 > 1e-13
0.038541666666666696 vs 0.03415488992713045, tolerance 2e-1 > 1e-13
0.45390624999999996 vs 0.4627803348994754, tolerance 2e-2 > 1e-13
0.9229166666666667 vs 0.9345951100728696, tolerance 2e-2 > 1e-13
**********************************************************************
File "/usr/lib/python3.7/site-packages/sage/plot/multigraphics.py", line 1302, in sage.plot.multigraphics.GraphicsArray.position
Failed example:
G.position(1) # tol 1.0e-13
Expected:
(0.5171874999999999,
0.038541666666666696,
0.45390624999999996,
0.9229166666666667)
Got:
(0.5136230468749999,
0.19293222169724827,
0.46278033489947534,
0.617040446532634)
Tolerance exceeded in 4 of 4:
0.5171874999999999 vs 0.5136230468749999, tolerance 7e-3 > 1e-13
0.038541666666666696 vs 0.19293222169724827, tolerance 5e0 > 1e-13
0.45390624999999996 vs 0.46278033489947534, tolerance 2e-2 > 1e-13
0.9229166666666667 vs 0.617040446532634, tolerance 4e-1 > 1e-13
**********************************************************************
1 item had failures:
2 of 6 in sage.plot.multigraphics.GraphicsArray.position
There may be more in 3.2 but I don't know of them yet.
I'm seeing one more:
sage -t --long --warn-long 99.2 src/sage/interacts/test_jupyter.rst
**********************************************************************
File "src/sage/interacts/test_jupyter.rst", line 280, in sage.interacts.test_jupyter
Failed example:
test(interacts.statistics.coin)
Expected:
Interactive function <function coin at ...> with 2 widgets
n: IntSlider(value=1000, description=u'Number of Tosses', max=10000, min=2, step=100)
interval: IntRangeSlider(value=(0, 0), description=u'Plotting range (y)', max=1)
doctest:...: UserWarning: Attempting to set identical bottom==top results
in singular transformations; automatically expanding.
bottom=0.0, top=0.0
Got:
Interactive function <function coin at 0x1505aab00> with 2 widgets
n: IntSlider(value=1000, description='Number of Tosses', max=10000, min=2, step=100)
interval: IntRangeSlider(value=(0, 0), description='Plotting range (y)', max=1)
doctest:warning
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/src/bin/sage-runtests", line 178, in <module>
err = DC.run()
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/doctest/control.py", line 1227, in run
self.run_doctests()
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/doctest/control.py", line 928, in run_doctests
self.dispatcher.dispatch()
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 2033, in dispatch
self.parallel_dispatch()
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1925, in parallel_dispatch
w.start() # This might take some time
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 2200, in start
super(DocTestWorker, self).start()
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 112, in start
self._popen = self._Popen(self)
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
return Popen(process_obj)
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
code = process_obj._bootstrap()
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 2172, in run
task(self.options, self.outtmpfile, msgpipe, self.result_queue)
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 2504, in __call__
doctests, extras = self._run(runner, options, results)
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 2553, in _run
result = runner.run(test)
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 897, in run
return self._run(test, compileflags, out)
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
exec(compiled, globs)
File "<doctest sage.interacts.test_jupyter[26]>", line 1, in <module>
test(interacts.statistics.coin)
File "<doctest sage.interacts.test_jupyter[3]>", line 6, in test
return f(**kwargs)
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/interacts/library.py", line 781, in coin
show(point(c[1:], gridlines=[None, [0.5]], pointsize=1), ymin=interval[0], ymax=interval[1])
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/repl/rich_output/pretty_print.py", line 252, in show
pretty_print(*args, **kwds)
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/repl/rich_output/pretty_print.py", line 223, in pretty_print
dm.display_immediately(*args, **kwds)
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py", line 837, in display_immediately
plain_text, rich_output = self._rich_output_formatter(obj, rich_repr_kwds)
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py", line 625, in _rich_output_formatter
rich_output = self._call_rich_repr(obj, rich_repr_kwds)
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py", line 583, in _call_rich_repr
return obj._rich_repr_(self, **rich_repr_kwds)
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/plot/graphics.py", line 1006, in _rich_repr_
self.save, kwds, file_ext, output_container)
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py", line 713, in graphics_from_save
save_function(filename, **kwds)
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/misc/decorators.py", line 412, in wrapper
return func(*args, **kwds)
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/plot/graphics.py", line 3315, in save
figure = self.matplotlib(**options)
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/sage/plot/graphics.py", line 2854, in matplotlib
subplot.set_ylim([ymin, ymax])
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/matplotlib/axes/_base.py", line 3604, in set_ylim
f"Attempting to set identical bottom == top == {bottom} "
File "/Users/palmieri/Desktop/Sage_stuff/git/sage/local/lib/python3.7/site-packages/matplotlib/cbook/__init__.py", line 1993, in _warn_external
warnings.warn(message, category, stacklevel)
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/warnings.py", line 110, in _showwarnmsg
msg.file, msg.line)
:
UserWarning: Attempting to set identical bottom == top == 0.0 results in singular transformations; automatically expanding.
**********************************************************************
1 item had failures:
1 of 30 in sage.interacts.test_jupyter
[29 tests, 1 failure, 6.43 s]
----------------------------------------------------------------------
sage -t --long --warn-long 99.2 src/sage/interacts/test_jupyter.rst # 1 doctest failed
Branch: u/jhpalmieri/matplotlib3.2.1
Description changed:
---
+++
@@ -1 +1,4 @@
Update to the latest matplotlib, currently version 3.2.1. This is Python 3-only, so it will be for Sage 9.2. This is necessary for upgrading Sphinx (#28856).
+
+File: https://files.pythonhosted.org/packages/4a/30/eb8e7dd8e3609f05c6920fa82f189302c832e5a0f6667aa96f952056bc0c/matplotlib-3.2.1.tar.gz
+Commit: 3e9f720
That one looks trivial to fix, like my second one. The first one will need more attention.
That should take care of the easy to fix doctests. I'll go back to that last one later if no ones beats me to it.
New commits:
84fb33c | Easy to fix doctests |
Changed branch from u/jhpalmieri/matplotlib3.2.1 to u/fbissey/matplotlib3.2.1
For the remaining doctest failure, I suggest deleting the doctest, or at least deleting the labels. I have some old versions of Sage on my computer, and going back to Sage 8.4 (py2) and 8.5 (py3), I see the same awful picture: see the attachment at #28449. The difference is now that matplotlib produces a warning in addition to producing the same terrible picture, so let's get rid of the warning by eliminating the plot labels.
Changed branch from u/fbissey/matplotlib3.2.1 to u/jhpalmieri/matplotlib3.2.1
This is ready for testing, I think.
New commits:
d21857c | trac 29547: fix doctest in root_lattice_realizations.py |
Author: John Palmieri, François Bissey
I was just looking into it. I like your stuff there. But someone familiar with that section of sage should revisit the problem because the code produce horrible stuff whether we test it or not. And someone will have to fix it or establish limits in which it is useful. I guess #28449 should be assigned to someone who can do that.
I just added a comment to #28449 about this. I think we should do this for now and let someone who knows the combinatorics part of things come up with a better fix.
Replying to @jhpalmieri:
I just added a comment to #28449 about this. I think we should do this for now and let someone who knows the combinatorics part of things come up with a better fix.
Agreed.
I have run ptestlong on vanilla sage 9.1.rc0 successfully - at least against stuff like matplotlib, my picked up system pari and ppl lead to some failures. But matplotlib-3.2.1 it is all good. We do mutual review of our changes or we find a third party?
@mkoeppe: when you have a chance, can you test this on some different platforms?
Replying to @kiwifb:
We do mutual review of our changes or we find a third party?
I'm pretty confident in it, and if we get it merged early in the 9.2 cycle, it will get lots of testing. So we could just review each other's changes.
Needs rebasing on top of current rc
@jhpalmieri, I am leaving the rebasing here to you as you are the current owner. I'll do #28856. MAy be add the upstream_url bit in matplotlib in this ticket.
The rebase will already give you the correct upstream_url for matplotlib
Now that the branch is on top of 9.1.rc1, all that is needed is to push it to a github repository and it will test on various systems
I am seeing a strange doctest failure on two OS X machines:
File "src/sage/schemes/elliptic_curves/ell_number_field.py", line 3944, in sage.schemes.elliptic_curves.ell_number_field.EllipticCurve_number_field.rational_points
Failed example:
E.rational_points(bound=5)
Exception raised:
Traceback (most recent call last):
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/generic/algebraic_scheme.py", line 1820, in rational_points
return X.points(**kwds) # checks for proper bound done in points functions
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/projective/projective_homset.py", line 275, in points
return sieve(X, B)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/projective/projective_rational_point.py", line 559, in sieve
rat_points = lift_all_points()
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/projective/projective_rational_point.py", line 537, in lift_all_points
lifted_points.add(X(tuple(point)))
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/generic/scheme.py", line 265, in __call__
return self.point(args)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/projective/projective_subscheme.py", line 126, in point
return self.point_homset()(v, check=check)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/generic/homset.py", line 275, in __call__
return Set_generic.__call__(self, *args, **kwds)
File "sage/structure/parent.pyx", line 902, in sage.structure.parent.Parent.__call__ (build/cythonized/sage/structure/parent.c:9245)
return mor._call_with_args(x, args, kwds)
File "sage/structure/coerce_maps.pyx", line 180, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_with_args (build/cythonized/sage/structure/coerce_maps.c:5081)
raise
File "sage/structure/coerce_maps.pyx", line 170, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_with_args (build/cythonized/sage/structure/coerce_maps.c:4871)
return C._element_constructor(x, **kwds)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/generic/homset.py", line 650, in _element_constructor_
return self.codomain()._point(self, v, **kwds)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/generic/algebraic_scheme.py", line 591, in _point
return self.__A._point(*args, **kwds)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/projective/projective_space.py", line 1327, in _point
return SchemeMorphism_point_projective_field(*args, **kwds)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/projective/projective_point.py", line 1063, in __init__
raise TypeError("v (=%s) must have %s components"%(v, d))
TypeError: v (=('N',)) must have 3 components
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
exec(compiled, globs)
File "<doctest sage.schemes.elliptic_curves.ell_number_field.EllipticCurve_number_field.rational_points[3]>", line 1, in <module>
E.rational_points(bound=Integer(5))
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/elliptic_curves/ell_number_field.py", line 3966, in rational_points
return [E(pt) for pt in Curve(self).rational_points(**kwds)]
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/generic/algebraic_scheme.py", line 1822, in rational_points
raise TypeError("Unable to enumerate points over %s."%F)
TypeError: Unable to enumerate points over Rational Field.
**********************************************************************
File "src/sage/schemes/elliptic_curves/ell_number_field.py", line 3946, in sage.schemes.elliptic_curves.ell_number_field.EllipticCurve_number_field.rational_points
Failed example:
E.rational_points(bound=6)
Exception raised:
Traceback (most recent call last):
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/generic/algebraic_scheme.py", line 1820, in rational_points
return X.points(**kwds) # checks for proper bound done in points functions
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/projective/projective_homset.py", line 275, in points
return sieve(X, B)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/projective/projective_rational_point.py", line 559, in sieve
rat_points = lift_all_points()
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/projective/projective_rational_point.py", line 537, in lift_all_points
lifted_points.add(X(tuple(point)))
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/generic/scheme.py", line 265, in __call__
return self.point(args)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/projective/projective_subscheme.py", line 126, in point
return self.point_homset()(v, check=check)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/generic/homset.py", line 275, in __call__
return Set_generic.__call__(self, *args, **kwds)
File "sage/structure/parent.pyx", line 902, in sage.structure.parent.Parent.__call__ (build/cythonized/sage/structure/parent.c:9245)
return mor._call_with_args(x, args, kwds)
File "sage/structure/coerce_maps.pyx", line 180, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_with_args (build/cythonized/sage/structure/coerce_maps.c:5081)
raise
File "sage/structure/coerce_maps.pyx", line 170, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_with_args (build/cythonized/sage/structure/coerce_maps.c:4871)
return C._element_constructor(x, **kwds)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/generic/homset.py", line 650, in _element_constructor_
return self.codomain()._point(self, v, **kwds)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/generic/algebraic_scheme.py", line 591, in _point
return self.__A._point(*args, **kwds)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/projective/projective_space.py", line 1327, in _point
return SchemeMorphism_point_projective_field(*args, **kwds)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/projective/projective_point.py", line 1063, in __init__
raise TypeError("v (=%s) must have %s components"%(v, d))
TypeError: v (=('N',)) must have 3 components
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
exec(compiled, globs)
File "<doctest sage.schemes.elliptic_curves.ell_number_field.EllipticCurve_number_field.rational_points[4]>", line 1, in <module>
E.rational_points(bound=Integer(6))
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/elliptic_curves/ell_number_field.py", line 3966, in rational_points
return [E(pt) for pt in Curve(self).rational_points(**kwds)]
File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc1/local/lib/python3.7/site-packages/sage/schemes/generic/algebraic_scheme.py", line 1822, in rational_points
raise TypeError("Unable to enumerate points over %s."%F)
TypeError: Unable to enumerate points over Rational Field.
**********************************************************************
1 item had failures:
2 of 10 in sage.schemes.elliptic_curves.ell_number_field.EllipticCurve_number_field.rational_points
This is strange because (a) I can't figure out what it has to do with matplotlib, and (b) tests only fail when run with --long, but the indicated tests are not marked # long time. The closest place there is a # long time tag is over 200 lines earlier.
Looks like pari/lcalc/ratpoints stuff to me.
Replying to @kiwifb:
Looks like pari/lcalc/ratpoints stuff to me.
Why would that be triggered by an upgrade to matplotlib?
Attachment: error_Couldnt_close_file.txt
Error message from running long tests by hand
I think we have a case where stuff may not be in matplotlib itself but some use of matplotlib may prime the memory for an error in another component to happen. In this case it is more a matter of highlighting a bug in that other component.
The strange bit that could be related to matplotlib is the use of freetype on the calls leading to the error. Is the stuff supposed to display something? If not why is it involved.
I don't know whether this info will be of any use, but I opened a sage session and ran the # long time tests in the file by hand (plus lines that defined necessary variables), and then ran the test that gives an error. It crashed (and I got a dialogue box that python had quit unexpectedly). The fundamental problem seems to be:
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Couldn't close file
My sage session survived, but seems to be pretty hosed. Now I get the crash even if I try E.rational_points(bound=1). An error log ("error_Couldnt_close_file.txt") is attached.
I did notice that the # long time tests on lines 2628, 2766, 2849, and 2855 involve G.show, which I suppose may involve matplotlib, so perhaps they have something to do with it.
That's good information and that definitely point to the possibility that there is some memory leakage leading to this.
If you have time. Does any single one of the long time doctests you highlighted enough to trigger the issue?
Here is a short sequence of commands that triggers a problem for me:
sage: K.<i> = QuadraticField(-1)
sage: E389 = EllipticCurve('389a1')
sage: EK = E389.change_ring(K)
sage: P = EK(-1,1); Q = EK(0,-1)
sage: EK.saturation([P+Q, 17*Q], lower_ht_bound=0.1)
([(4 : 8 : 1), (0 : -1 : 1)], 17, 0.152460177943143)
sage: E2 = EllipticCurve(K, [0,0,0,0,1])
sage: C = E2.isogeny_class()
sage: G = C.graph()
sage: G.show()
Launched png viewer for Graphics object consisting of 9 graphics primitives
sage: E = EllipticCurve('11a1')
sage: E.rational_points(bound=5)
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Couldn't close file
------------------------------------------------------------------------
0 signals.cpython-37m-darwin.so 0x000000010ef05f22 print_backtrace + 66
1 signals.cpython-37m-darwin.so 0x000000010ef0a1a7 sigdie + 39
2 signals.cpython-37m-darwin.so 0x000000010ef0a120 sigdie_for_sig + 256
3 libsystem_platform.dylib 0x00007fff7367d5fd _sigtramp + 29
...
And here is the one that I found:
pol26 = hilbert_class_polynomial(-4*26)
pol = x^6-x^5+2*x^4+x^3-2*x^2-x-1
K.<a> = NumberField(pol)
j1 = pol26.roots(K)[0][0]
E = EllipticCurve(j=j1)
C = E.isogeny_class(); len(C) # long time (~11s)
G = C.graph()
G.show(edge_labels=False) # long time
E = EllipticCurve("11a1")
E.rational_points(bound=5)
E.rational_points(bound=5)
Note that the last command needs to be done twice. For me (9.1.rc1 on Mac OS 10.15.3), it works the first time, but crashes when it runs again.
The commands from jhpalmieri don't give me an exception unless I do the last command twice (like in mine).
sage: K.<i> = QuadraticField(-1)
sage: E389 = EllipticCurve('389a1')
sage: EK = E389.change_ring(K)
sage: P = EK(-1,1); Q = EK(0,-1)
sage: EK.saturation([P+Q, 17*Q], lower_ht_bound=0.1)
([(4 : 8 : 1), (0 : -1 : 1)], 17, 0.152460177943143)
sage: E2 = EllipticCurve(K, [0,0,0,0,1])
sage: C = E2.isogeny_class()
sage: G = C.graph()
sage: G.show()
Launched png viewer for Graphics object consisting of 9 graphics primitives
sage: E = EllipticCurve('11a1')
sage: E.rational_points(bound=5)
[(0 : 1 : 0), (5 : 5 : 1)]
sage: E.rational_points(bound=5)
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Couldn't close file
PS bound=3 gives me the same behavior as bound=5, but bound=2 gives no error until I run it a third time.
sage: K.<i> = QuadraticField(-1)
sage: E389 = EllipticCurve('389a1')
sage: EK = E389.change_ring(K)
sage: P = EK(-1,1); Q = EK(0,-1)
sage: EK.saturation([P+Q, 17*Q], lower_ht_bound=0.1)
([(4 : 8 : 1), (0 : -1 : 1)], 17, 0.152460177943143)
sage: E2 = EllipticCurve(K, [0,0,0,0,1])
sage: C = E2.isogeny_class()
sage: G = C.graph()
sage: G.show()
Launched png viewer for Graphics object consisting of 9 graphics primitives
sage: E = EllipticCurve('11a1')
sage: E.rational_points(bound=2)
[(0 : 1 : 0)]
sage: E.rational_points(bound=2)
[(0 : 1 : 0)]
sage: E.rational_points(bound=2)
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Couldn't close file
Let me make two unhelpful comments:
- many of the doctests marked as
# long timein this file don't actually take a long time. Maybe they used to, but perhaps algorithms have changed, and now they don't. - the plot produced in
G = C.graph()andG.show()is very simple: 4 vertices and 4 edges arranged in a square. So it's not clear whymatplotlibis triggering the problem here. It certainly seems to be, but I don't understand it. Maybe the problem is fundamentally somewhere else, and we're just seeing it now.
I fairly convinced the update just made a memory leak visible. Either in freetype (which has just been updated) or pari.
I'm using homebrew's freetype, so I'm guessing that's not it. I'll try building with ./configure --with-system-freetype=no and see what happens.
Here are a couple of short crashers that use the same elliptic curve for both parts:
sage: E = EllipticCurve("11a1")
sage: C = E.isogeny_class()
sage: G = C.graph()
sage: G.show(edge_labels=False)
Launched png viewer for Graphics object consisting of 6 graphics primitives
sage: E.rational_points(bound=5)
[(0 : 1 : 0), (5 : 5 : 1)]
sage: E.rational_points(bound=5)
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Couldn't close file
sage: E = EllipticCurve('389a1')
sage: C = E.isogeny_class()
sage: G = C.graph()
sage: G.show(edge_labels=False)
Launched png viewer for Graphics object consisting of 2 graphics primitives
sage: E.rational_points(bound=5)
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Couldn't close file
In the second one, the graph displayed by G.show() is a single vertex.
Here is a much simpler example that I hope will help narrow down the issue. I get an exception even if the picture drawn by matplotlib has nothing to do with elliptic curves.
sage: text("hi", (0,0)).show(axes=False)
Launched png viewer for Graphics object consisting of 1 graphics primitive
sage: E = EllipticCurve('389a1')
sage: E.rational_points(bound=5);
sage: E.rational_points(bound=5);
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Couldn't close file
I also get a crash when I change text("hi", (0,0)).show(axes=False) to Graphics().show() (which only has the x-axis and y-axis, with labels), but not when I change it to circle((1,1),1).show(axes=False).
I think all known examples that cause the problem have text in the picture.
There is no crash (for me) if latex typesets the text. This seems to support the suggestion of fbissey in comment:36 that the real culprit may be freetype, rather than matplotlib itself.
sage: from matplotlib import rc
sage: rc("text", usetex=True)
sage: text("hi", (0,0)).show(axes=False)
Launched png viewer for Graphics object consisting of 1 graphics primitive
sage: E = EllipticCurve('389a1')
sage: E.rational_points(bound=5);
sage: E.rational_points(bound=5);
sage: E.rational_points(bound=5);
I wrote a loop to repeat the last line 100 times and did not get a crash.
I tried upgrading to freetype 2.10.2, the latest, and I still see the doctest failures.
That's annoying. For those who know freetype well, or perhaps how matplotlib connects to it, this is presumably where we have some leakage
12 ft2font.cpython-37m-darwin.so 0x000000039a95d96d _ZL19close_file_callbackP13FT_StreamRec_ + 541
13 libfreetype.6.dylib 0x00000003912e4567 destroy_face + 234
14 libfreetype.6.dylib 0x00000003912e4427 FT_Done_Face + 153
15 ft2font.cpython-37m-darwin.so 0x000000039a954ef1 _ZN7FT2FontD2Ev + 97
16 ft2font.cpython-37m-darwin.so 0x000000039a954fcf _ZN7FT2FontD0Ev + 15
17 ft2font.cpython-37m-darwin.so 0x000000039a95c668 _ZL17PyFT2Font_deallocP9PyFT2Font + 24
The deallocation probably doesn't, or leave some objects behind. And on some machines we have a boom when that corrupted memory is accessed.
I think we may have this matplotlib/matplotlib#15104 in which case we may try to adopt the patch if we can.
Ok technically the bug report is in matplotlib/matplotlib#15410 but the fix for it is in the pull above.
This looks promising: tests pass on the file ell_number_field.py. I'm running full tests now. François, can you make a new branch?
Replying to @jhpalmieri:
This looks promising: tests pass on the file ell_number_field.py. I'm running full tests now. François, can you make a new branch?
For the record, I cannot reproduce the problem on my machine. I'll prepare a branch ASAP.
Would it be any easier for distro people if Sage applies the patch only on OS X?
Replying to @jhpalmieri:
Would it be any easier for distro people if Sage applies the patch only on OS X?
I wouldn't think so. If anything distros that see cases may want to adopt the patch. I am thinking of gentoo in prefix on OS X and conda (because it ships on OS X) in particular.
Generally speaking, this patch doesn't affect functionality or behavior in sage. So, from the point of view of distros it is safe. It only deals with a corner case that is not just a sage problem.
Branch done.
New commits:
bb7e259 | Add matplotlib upstream patch to deal with memory leaks on OS X. The patch should probably removed in the next matplotlib upgrade. |
Changed branch from u/jhpalmieri/matplotlib3.2.1 to u/fbissey/matplotlib3.2.1
This branch eliminates all of the crashes that I saw before. Thanks!
Replying to @DaveWitteMorris:
This branch eliminates all of the crashes that I saw before. Thanks!
Glad to hear it! I guess we are close to have a positive review now.
Ideally, we would want this ticket to be in the first load of merges towards 9.2.beta0 so we can can proceed quickly with #28856 and have sphinx 3 in beta0 or beta1 at the latest. I and John are authors. Who will volunteer to put their name in the reviewer field towards a positive review.
We also have to remember that this ticket will break py2 compatibility. And we will have many of that kind asking to be merged very fast.
In my opinion, it's a pretty straightforward update of a standard package. The new patch is the main complicating factor, but that's been accepted upstream already.
Now that the upstream patch has been applied, I agree that this ticket seems straightforward. (All issues have been addressed, and the temporary fix mentioned in comment:10 will be remedied when I upload a PR to #28449.) However, the patchbots do not seem to be running on this (comment:24 says something about testing on various systems, but I haven't seen any results from that), and I don't feel I have enough sagemath experience yet to give a positive review on a substantial ticket. (My PRs often need minor corrections.)
comment:18 suggested that you might review each other's changes. If that is a possibility, then I would endorse it in this case. As an alternative, I would be willing to put my name in the reviewer field if fbissey gives an explicit endorsement of this ticket (I take comment:54 as an endorsement from jhpalmieri), and somebody explains to me why it is ok not to have any test results. But it's great if someone with more experience than me wants to step in.
I think it is best to have at least one other person in the reviewer fied as it show some more people have looked at it. It is not completely untested although not the traditional way you may expect. Both sage-on-gentoo and sage-on-arch ship a variant of this already. And there will be another round of CI before it is finally merged that could catch stuff we haven't already, especially on platform like cygwin.
I looked more carefully at what is in the commits, and it all looks good to me. Thanks, both of you, for your work on this! I am setting to positive review.
Reviewer: Dave Morris
Changed branch from u/fbissey/matplotlib3.2.1 to bb7e259
On OS Python 3.5:
****************************************************
Package 'matplotlib' is currently not installed
No legacy uninstaller found for 'matplotlib'; nothing to do
NOTE: Set SAGE_MATPLOTLIB_GUI to anything but 'no' to try to build the Matplotlib GUI.
Not building any matplotlib graphical backends.
Installing matplotlib-3.2.1
File "setup.py", line 139
raise IOError(f"Failed to download jquery-ui. Please download "
^
SyntaxError: invalid syntax
Error: could not determine package name
********************************************************************************
Error installing matplotlib-3.2.1
********************************************************************************
What exactly are our new system requirements? I posted on sage-devel for clarification
Should we make this change?
diff --git a/build/pkgs/matplotlib/spkg-install.in b/build/pkgs/matplotlib/spkg-install.in
index 6b68b46ad5..a427dadccd 100644
--- a/build/pkgs/matplotlib/spkg-install.in
+++ b/build/pkgs/matplotlib/spkg-install.in
@@ -1,5 +1,5 @@
# Write a configuration file to src/setup.cfg
-sage-system-python make-setup-config.py
+sage-python23 make-setup-config.py
cd src
Then we would be using Sage's Python to build here. This should be okay, since Python is a dependency for matplotlib.
Some Sage packages use sage-system-python, some use sage-python23. Some of the ones using sage-system-python are build before Python, which makes sense, but for packages like matplotlib, I don't see any reason to not use sage-python23.
Changed branch from bb7e259 to u/jhpalmieri/bb7e259951c508c3279ee02756e12914a911b7fa
Commit: eee7843
New commits:
9a4b89b | trac 29541: matplotlib 3.2.1 |
cad0196 | Easy to fix doctests |
8640711 | trac 29547: fix doctest in root_lattice_realizations.py |
b436dbc | trac 29547: rebase and restore upstream_url line in checksums.ini |
bb7e259 | Add matplotlib upstream patch to deal with memory leaks on OS X. The patch should probably removed in the next matplotlib upgrade. |
eee7843 | trac 29547: use sage-python23 instead of sage-system-python to install |
There should be just one new commit, the last one in that list. We've already seen the rest.
Using sage-python23 (which could, actually, be replaced by python3 -- but this simplification should be done uniformly on another ticket) is the correct solution here. (See discussion in https://groups.google.com/d/msg/sage-devel/fgsSsJmVVXo/Taix5GaQAAAJ)
Changed reviewer from Dave Morris to Dave Morris, Matthias Koeppe
Changed branch from u/jhpalmieri/bb7e259951c508c3279ee02756e12914a911b7fa to eee7843