doctests: Detect "safe" external features even if "--optional=external" is not used

Question

doctests: Detect "safe" external features even if "--optional=external" is not used

Closed this issue 4 years ago · 117 comments

For some "safe" features such as imagemagick, rubiks, 4ti2 (#30887), pandoc, it really shouldn't require the use of external to check for these.

Likewise for the PythonModule features for Sage modules introduced in #32614. By delaying the test for these features, using the sage.doctest.external machinery, we fix the broken detection of some of these features. (This is why the ticket is marked "critical".)

This is a step toward #30746 (sage.doctest.control: Replace use of sage.misc.package.list_packages)

With this ticket, more doctests will be run by default. This may lead to new failures (discovered by patchbot for instance). But that does not mean it is introduced in this ticket, just that they were existing before this ticket, but simply not executed.

For example, let's compare the behavior of the following command:

sage -tp --show-skipped src/sage/tests/cmdline.py

returns:

BEFORE (9.5.beta7):

Git branch: develop
Using --optional=4ti2,build,ccache,cryptominisat,debian,dochtml,dot2tex,e_antic,fricas,glucose,latte_int,lidia,normaliz,notedown,pandoc_attributes,pip,pycosat,pynormaliz,rst2ipynb,sage,sage.geometry.polyhedron,sage.rings.real_double,sage_numerical_backends_coin,sage_spkg
Doctesting 1 file using 8 threads.
sage -t --random-seed=265206128165572261397002909756599539048 src/sage/tests/cmdline.py
    3 gdb tests not run
    12 internet tests not run
    2 pandoc tests not run
    4 r tests not run
    0 tests not run because we ran out of time
    [235 tests, 30.65 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------

AFTER (9.5.beta7 + #32174):

$ sage -tp --show-skipped src/sage/tests/cmdline.py 
Git branch: 32174
Using --optional=4ti2,build,ccache,cryptominisat,debian,dochtml,dot2tex,e_antic,fricas,glucose,latte_int,lidia,normaliz,notedown,pandoc_attributes,pip,pycosat,pynormaliz,rst2ipynb,sage,sage_numerical_backends_coin,sage_spkg
Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,ffmpeg,graphviz,imagemagick,jupymake,kenzo,latte_int,lrslib,mcqd,meataxe,pandoc,pdf2svg,plantri,pynormaliz,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,tdlib
Doctesting 1 file using 8 threads.
sage -t --random-seed=100100306103974678574881156032164018710 src/sage/tests/cmdline.py
**********************************************************************
File "src/sage/tests/cmdline.py", line 732, in sage.tests.cmdline.test_executable
Failed example:
    print(open(output, 'r').read() == t)          # optional - pandoc
Expected:
    True
Got:
    False
**********************************************************************
1 item had failures:
   1 of 238 in sage.tests.cmdline.test_executable
    3 gdb tests not run
    12 internet tests not run
    4 r tests not run
    0 tests not run because we ran out of time
    [237 tests, 1 failure, 32.76 s]
----------------------------------------------------------------------
sage -t --random-seed=100100306103974678574881156032164018710 src/sage/tests/cmdline.py  # 1 doctest failed
----------------------------------------------------------------------
Features detected for doctesting: pandoc

We observe that the number of tests run went from 235 to 237, because the feature pandoc was detected. Note that the doctest failure is already tracked at #32697.

Depends on #32866
Depends on #27744

CC: @dimpase @slel @antonio-rojas @kiwifb @kwankyu @seblabbe @videlec

Component: doctest framework

Author: Matthias Koeppe, Sébastien Labbé

Branch: 41c4cd4

Reviewer: Sébastien Labbé

Issue created by migration from https://trac.sagemath.org/ticket/32174

Answer 1 · 2021-10-02T23:40:02.000Z

Description changed:

--- 
+++ 
@@ -2,5 +2,7 @@
 
 For some "safe" features such as `imagemagick`, `rubiks`, `4ti2` (#30887), `pandoc`, it really shouldn't require the use of `external` to check for these. 
 
+Likewise for the `PythonModule` features for Sage modules introduced in #32614.
+
 This is a step toward #30746 (`sage.doctest.control`: Replace use of `sage.misc.package.list_packages`)

Answer 2 · 2021-10-07T18:34:37.000Z

Dependencies: #32649

Answer 3 · 2021-10-07T23:40:20.000Z

Changed dependencies from #32649 to #32649, #32614

Answer 4 · 2021-10-22T20:06:53.000Z

comment:5

Let's use this ticket to address the problem discovered in #32732 comment:10: Detection of Sage modules fails because the check runs before sage.all has been imported, and is therefore affected by import failures from cyclic imports

Answer 5 · 2021-10-22T20:59:21.000Z

Changed dependencies from #32649, #32614 to #32732

Answer 6 · 2021-11-08T05:38:45.000Z

Changed dependencies from #32732 to #32732, #32650

Answer 7 · 2021-11-11T02:06:13.000Z

Branch: u/mkoeppe/doctests__detect__safe__external_features_even_if____optional_external__is_not_used

Answer 8 · 2021-11-11T02:07:40.000Z

comment:9

Here's an approach to do feature enumeration

Last 10 new commits:

`5d9414d`	`32650: .. note -> .. NOTE`
`345c760`	`32650: using features in has_latex method of doctest/external.py`
`6c6b332`	`32650: independent and improved method latex().is_functional()`
`63555d2`	`32650: adding spkg pdf2svg in build/pkgs`
`c8330fe`	`32650: linking pdf2svg feature to spkg pdf2svg`
`2ea4d9e`	`32650: fixing two doctests now that we latex().require() it when we use it`
`98b7db6`	`Merge #32650`
`0968f95`	`sage.features.sagemath.all_features: New`
`cdc0cb6`	`sage.features.all: New`
`c8e6c0e`	`sage.features.bliss.all_features: New`

Answer 9 · 2021-11-11T02:07:40.000Z

Commit: c8e6c0e

Answer 10 · 2021-11-11T03:10:55.000Z

Changed commit from c8e6c0e to 69d2cfb

Answer 11 · 2021-11-11T03:10:55.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`69d2cfb`	`src/sage/features/kenzo.py: Move import from sage.libs.ecl into method`

Answer 12 · 2021-11-11T04:49:00.000Z

Author: Matthias Koeppe, ...

Answer 13 · 2021-11-13T13:03:31.000Z

comment:12

Indent .. NOTE:: blocks 4 spaces, not 3
(can be a follow-up ticket).

Answer 14 · 2021-11-13T18:18:28.000Z

Changed commit from 69d2cfb to 893ec23

Answer 15 · 2021-11-13T18:18:28.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`fa45c36`	`sage.features.sagemath.all_features: New`
`17e30e2`	`sage.features.all: New`
`8bfebc9`	`sage.features.bliss.all_features: New`
`893ec23`	`src/sage/features/kenzo.py: Move import from sage.libs.ecl into method`

Answer 16 · 2021-11-13T19:50:21.000Z

Changed dependencies from #32732, #32650 to #32866

Answer 17 · 2021-11-13T19:50:21.000Z

comment:14

Rebased on 9.5.beta6

Answer 18 · 2021-11-14T22:04:47.000Z

Changed commit from 893ec23 to fa25155

Answer 19 · 2021-11-14T22:04:47.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`eaec400`	`src/sage/features/interfaces.py: New`
`827ca03`	`src/sage/features/interfaces.py: Fix and doctest exception handling`
`52ac51e`	`src/sage/features/interfaces.py (all_features): New`
`7754fae`	`src/sage/features/interfaces.py: Fix and doctest exception handling (fixup)`
`8081f29`	`src/sage/doctest/external.py: Use sage.features.interfaces`
`09c4238`	`Merge #32866`
`c41808b`	`src/sage/features/latex.py (all_features): New`
`44c19e3`	`src/sage/features/mip_backends.py (all_features): New`
`fa25155`	`src/sage/doctest/external.py (external_software): Go through external_features`

Answer 20 · 2021-11-14T22:25:07.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`945132b`	`src/sage/features/interfaces.py: Fix up`
`f63acfd`	`Merge #32866`
`7fa5b3b`	`src/sage/doctest/external.py (external_features): Add CPLEX, Gurobi`

Answer 21 · 2021-11-14T22:25:07.000Z

Changed commit from fa25155 to 7fa5b3b

Answer 22 · 2021-11-14T22:27:53.000Z

comment:17

The has_... functions are now no longer needed.

Deprecate them? Or just remove them?

Answer 23 · 2021-11-14T23:43:39.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`132f00d`	`sage.doctest.{control,external}: Detect optional tags via AvailableSoftware`

Answer 24 · 2021-11-14T23:43:39.000Z

Changed commit from 7fa5b3b to 132f00d

Answer 25 · 2021-11-14T23:44:47.000Z

comment:19

Here's a version that seems to work.

Answer 26 · 2021-11-15T00:08:31.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`68b3d41`	`src/sage/doctest/external.py (external_features): Rewrite as generator, add doctest`

Answer 27 · 2021-11-15T00:08:31.000Z

Changed commit from 132f00d to 68b3d41

Answer 28 · 2021-11-15T04:26:06.000Z

Changed author from Matthias Koeppe, ... to Matthias Koeppe

Answer 29 · 2021-11-15T07:38:09.000Z

comment:22

Is the ticket description still valid? This doesn't do anything for me unless I explicitly pass --optional=external

Answer 30 · 2021-11-15T17:34:55.000Z

comment:23

At the end of the tests it should print a line "Features detected for doctesting"

Answer 31 · 2021-11-15T17:42:19.000Z

comment:24

Yes, but it doesn't detect anything if I don't use --optional=external:

> sage -t --optional=sage /usr/lib/python3.9/site-packages/sage/features/imagemagick.py
too many failed tests, not using stored timings
Running doctests with ID 2021-11-15-18-40-01-8b3be5fc.
Using --optional=sage
External software to be detected: 4ti2,cplex,dvipng,ffmpeg,graphviz,gurobi,imagemagick,internet,latex,lualatex,macaulay2,magma,maple,mathematica,matlab,octave,pandoc,pdf2svg,pdflatex,rubiks,scilab,xelatex
Doctesting 1 file.
sage -t --random-seed=283871835415461728632029979826455200075 /usr/lib/python3.9/site-packages/sage/features/imagemagick.py
    [3 tests, 0.00 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 0.0 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 0.0 seconds
Features detected for doctesting: 
Pytest is not installed, skip checking tests that rely on it.

doesn't detect imagemagick, but

> sage -t --optional=sage,external /usr/lib/python3.9/site-packages/sage/features/imagemagick.py
too many failed tests, not using stored timings
Running doctests with ID 2021-11-15-18-39-46-a52ab438.
Using --optional=external,sage
External software to be detected: 4ti2,cplex,dvipng,ffmpeg,graphviz,gurobi,imagemagick,internet,latex,lualatex,macaulay2,magma,maple,mathematica,matlab,octave,pandoc,pdf2svg,pdflatex,rubiks,scilab,xelatex
Doctesting 1 file.
sage -t --random-seed=59814167025539044529043021967633197695 /usr/lib/python3.9/site-packages/sage/features/imagemagick.py
    [4 tests, 0.00 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 0.0 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 0.0 seconds
Features detected for doctesting: imagemagick
Pytest is not installed, skip checking tests that rely on it.

does

Answer 32 · 2021-11-15T17:47:53.000Z

Changed commit from 68b3d41 to 6f2d22b

Answer 33 · 2021-11-15T17:47:53.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`6f2d22b`	`src/sage/doctest/external.py: Declare pandoc, graphviz, ffmpeg, imagemagick, dvipng, pdf2svg, rubiks, 4ti2 feature tests safe (do not require 'external' to test them)`

Answer 34 · 2021-11-15T17:49:30.000Z

comment:26

Thanks. Should work now

Answer 35 · 2021-11-15T18:04:34.000Z

Changed commit from 6f2d22b to 2f50385

Answer 36 · 2021-11-15T18:04:34.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`2f50385`	`src/sage/features/: Add more all_features functions`

Answer 37 · 2021-11-15T18:18:02.000Z

Changed commit from 2f50385 to e2cda41

Answer 38 · 2021-11-15T18:18:02.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`e2cda41`	`src/sage/features/: Add remaining missing all_features functions`

Answer 39 · 2021-11-15T18:21:42.000Z

comment:29

Replying to @mkoeppe:

Thanks. Should work now

Confirmed. But now, running, sage -t without --optional=external displays

External software to be detected: cplex,gurobi,internet,macaulay2,magma,maple,mathematica,matlab,octave,scilab

which is kind of misleading, as these are precisely the packages that are not autodetected in that case.

Answer 40 · 2021-11-15T18:29:33.000Z

comment:30

Thanks for testing. Good point, I'll make that clearer

Answer 41 · 2021-11-15T18:43:59.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`c4deceb`	`src/sage/doctest/control.py: Replace use of 'external_software' by new method 'available_software.detectable()'`

Answer 42 · 2021-11-15T18:43:59.000Z

Changed commit from e2cda41 to c4deceb

Answer 43 · 2021-11-15T22:25:54.000Z

Changed dependencies from #32866 to #32866, #27744

Answer 44 · 2021-11-15T22:27:04.000Z

Changed commit from c4deceb to b8f0a10

Answer 45 · 2021-11-15T22:27:04.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`23f5009`	`sage.features.normaliz: New`
`0b65b79`	`src/sage/features/normaliz.py: Fix up`
`00cb9c7`	`src/sage/features/normaliz.py: Rename class to PyNormaliz, fix uses`
`7632e53`	`src/sage/geometry/polyhedron/base.py: Remove unused import`
`24aaf73`	`Merge #27744`
`b8f0a10`	`src/sage/features/normaliz.py: Add all_features`

Answer 46 · 2021-11-16T00:46:58.000Z

Changed commit from b8f0a10 to 91f0bcc

Answer 47 · 2021-11-16T00:46:58.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`944bb74`	`src/sage/features/interfaces.py: Add doctests`
`2463e1e`	`Merge #32866`
`91f0bcc`	`src/sage/doctest: Update doctest outputs`

Answer 48 · 2021-11-16T01:06:10.000Z

Changed commit from 91f0bcc to c9312db

Answer 49 · 2021-11-16T01:06:10.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`12f7c98`	`src/sage/features/polymake.py: Add all_features`
`c9312db`	`src/sage/features: Add more all_features functions`

Answer 50 · 2021-11-16T16:21:09.000Z

comment:36

This unveils a test failure coming from #32732 (wasn't detected there because the test was skipped)

**********************************************************************
File "/usr/lib/python3.9/site-packages/sage/geometry/hyperbolic_space/hyperbolic_geodesic.py", line 2038, in sage.geometry.hyperbolic_space.hyperbolic_geodesic.HyperbolicGeodesicPD.plot
Failed example:
    H = h.plot(thickness=6, color="orange"); H           # optional - sage.plot
Expected:
    Graphics object consisting of 2 graphics primitives        # optional - sage.plot
Got:
    Graphics object consisting of 2 graphics primitives
**********************************************************************

Answer 51 · 2021-11-16T17:21:58.000Z

Changed commit from c9312db to 230d135

Answer 52 · 2021-11-16T17:21:58.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`230d135`	`src/sage/geometry/hyperbolic_space/hyperbolic_geodesic.py: Fix doctest markup`

Answer 53 · 2021-11-16T17:25:55.000Z

Description changed:

--- 
+++ 
@@ -2,7 +2,7 @@
 
 For some "safe" features such as `imagemagick`, `rubiks`, `4ti2` (#30887), `pandoc`, it really shouldn't require the use of `external` to check for these. 
 
-Likewise for the `PythonModule` features for Sage modules introduced in #32614.
+Likewise for the `PythonModule` features for Sage modules introduced in #32614. By delaying the test for these features, using the `sage.doctest.external` machinery, we fix the broken detection of some of these features. (This is why the ticket is marked "critical".)
 
 This is a step toward #30746 (`sage.doctest.control`: Replace use of `sage.misc.package.list_packages`)

Answer 54 · 2021-11-16T19:42:11.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`7a8f48c`	`src/sage/features/sagemath.py (sage_features): Unused now, removed`

Answer 55 · 2021-11-16T19:42:11.000Z

Changed commit from 230d135 to 7a8f48c

Answer 56 · 2021-11-17T19:29:58.000Z

comment:40

I'm getting a bliss related failure:

File "/usr/lib/python3.9/site-packages/sage/features/bliss.py", line 33, in sage.features.bliss.BlissLibrary
Failed example:
    BlissLibrary().require()  # optional: bliss
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.9/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.9/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.features.bliss.BlissLibrary[1]>", line 1, in <module>
        BlissLibrary().require()  # optional: bliss
      File "/usr/lib/python3.9/site-packages/sage/features/__init__.py", line 205, in require
        raise FeatureNotPresentError(self, presence.reason, presence.resolution)
    sage.features.FeatureNotPresentError: Bliss is not available.
    Failed to run test code.
    No equivalent system packages for pip are known to Sage.
    Further installation instructions might be available at http://www.tcs.hut.fi/Software/bliss/.

This is happening because Sage is compiled with bliss, but the bliss library is not present at runtime (it doesn't need to, since it's statically linked).

So apparently there is a mismatch between whatever is tested to decide whether to enable the optional: bliss tests (which seems to be just the presence of the python module) and what the tests themselves require (not just the module, but also the library itself)

Answer 57 · 2021-11-17T19:42:23.000Z

comment:41

Also, pyflakes patchbot says:

src/sage/doctest/forker.py:218:9 'readline' imported but unused
src/sage/doctest/forker.py:224:9 'sympy' imported but unused
src/sage/geometry/polyhedron/base.py:2701:17 'sage.graphs.graph' imported but unused
src/sage/geometry/polyhedron/base.py:5852:17 'sage.graphs.graph' imported but unused
src/sage/geometry/polyhedron/base.py:6438:17 'sage.graphs.graph' imported but unused
src/sage/geometry/polyhedron/base.py:8913:13 'sage.graphs.graph' imported but unused

Answer 58 · 2021-11-17T19:44:28.000Z

comment:42

Also another plugin patchbot warning says:

wrong syntax for blocks (INPUT, OUTPUT, EXAMPLES, NOTE, etc) inserted on 9 non-empty lines

possibly in

inside file:  b/src/sage/features/interfaces.py
inside file:  b/src/sage/features/normaliz.py

Answer 59 · 2021-11-17T21:01:12.000Z

comment:43

Replying to @seblabbe:

Also, pyflakes patchbot says:

src/sage/doctest/forker.py:218:9 'readline' imported but unused
src/sage/doctest/forker.py:224:9 'sympy' imported but unused
src/sage/geometry/polyhedron/base.py:2701:17 'sage.graphs.graph' imported but unused
src/sage/geometry/polyhedron/base.py:5852:17 'sage.graphs.graph' imported but unused
src/sage/geometry/polyhedron/base.py:6438:17 'sage.graphs.graph' imported but unused
src/sage/geometry/polyhedron/base.py:8913:13 'sage.graphs.graph' imported but unused

Not from this ticket, and all of these imports are done for their side effect, not the binding

Answer 60 · 2021-11-17T21:06:45.000Z

Changed commit from 7a8f48c to 12a2507

Answer 61 · 2021-11-17T21:06:45.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`e7f538d`	`sage.features.bliss: Change name of BlissLibrary feature to libbliss, remove it from all_features`
`12a2507`	`src/sage/features/interfaces.py: Fix indentation in docstring`

Answer 62 · 2021-11-18T14:15:46.000Z

comment:45

For reviewing, this ticket, I did:

$ sage -t -p --all --long --logfile=logs/32174-second.log

which returned

Git branch: 32174
Using --optional=4ti2,build,ccache,cryptominisat,debian,dochtml,dot2tex,e_antic,fricas,glucose,latte_int,lidia,normaliz,notedown,pandoc_attributes,pip,pycosat,pynormaliz,rst2ipynb,sage,sage_numerical_backends_coin,sage_spkg
Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,ffmpeg,graphviz,imagemagick,jupymake,kenzo,latex,latte_int,lrslib,lualatex,mcqd,meataxe,pandoc,pdf2svg,pdflatex,plantri,pynormaliz,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,tdlib,xelatex
Doctesting entire Sage library.
Sorting sources by runtime so that slower doctests are run first....
Doctesting 4397 files using 8 threads.
...
----------------------------------------------------------------------
sage -t --long --random-seed=129857008849586417489695899017012168188 src/sage/tests/cmdline.py  # 1 doctest failed
sage -t --long --random-seed=129857008849586417489695899017012168188 src/sage/graphs/generic_graph.py  # 2 doctests failed
sage -t --long --random-seed=129857008849586417489695899017012168188 src/sage/interfaces/fricas.py  # 1 doctest failed
sage -t --long --random-seed=129857008849586417489695899017012168188 src/sage/graphs/graph.py  # 2 doctests failed
sage -t --long --random-seed=129857008849586417489695899017012168188 src/sage/schemes/toric/sheaf/klyachko.py  # 1 doctest failed
sage -t --long --random-seed=129857008849586417489695899017012168188 src/sage/sat/solvers/dimacs.py  # 2 doctests failed
sage -t --long --random-seed=129857008849586417489695899017012168188 src/sage/combinat/designs/incidence_structures.py  # 1 doctest failed
----------------------------------------------------------------------
Features detected for doctesting: dvipng,ffmpeg,graphviz,imagemagick,latex,lualatex,pandoc,pdf2svg,pdflatex,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,xelatex

The issue with src/sage/schemes/toric/sheaf/klyachko.py does not reproduce:

sage -t --long --random-seed=129857008849586417489695899017012168188 src/sage/schemes/toric/sheaf/klyachko.py
**********************************************************************
File "src/sage/schemes/toric/sheaf/klyachko.py", line 950, in sage.schemes.toric.sheaf.klyachko.KlyachkoBundle_clas
s.random_deformation
Failed example:
    Vtilde.cohomology(dim=True, weight=(0,))
Expected:
    (1, 0)
Got:
    (0, 0)
**********************************************************************
1 item had failures:
   1 of   7 in sage.schemes.toric.sheaf.klyachko.KlyachkoBundle_class.random_deformation
    [151 tests, 1 failure, 16.00 s]

The other failures are known.

Then I did:

sage -t -p --all --long --optional=sage,optional,external,build --show-skipped --logfile=logs/32174.log

which returned:

Git branch: 32174
Using --optional=4ti2,build,ccache,cryptominisat,debian,dot2tex,e_antic,external,fricas,glucose,latte_int,lidia,normaliz,notedown,pandoc_attributes,pip,pycosat,pynormaliz,rst2ipynb,sage,sage_numerical_backends_coin,sage_spkg
Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,cplex,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,ffmpeg,graphviz,gurobi,imagemagick,internet,jupymake,kenzo,latex,latte_int,lrslib,lualatex,macaulay2,magma,maple,mathematica,matlab,mcqd,meataxe,octave,pandoc,pdf2svg,pdflatex,plantri,pynormaliz,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,scilab,tdlib,xelatex
Doctesting entire Sage library.
Sorting sources by runtime so that slower doctests are run first....
Doctesting 4397 files using 8 threads.
...
----------------------------------------------------------------------
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/graphs/generic_graph.py  # 2 doctests failed
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/databases/findstat.py  # 9 doctests failed
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/tests/cmdline.py  # 10 doctests failed
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/graphs/graph.py  # 3 doctests failed
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/interfaces/fricas.py  # 1 doctest failed
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/sat/solvers/dimacs.py  # 2 doctests failed
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/combinat/designs/incidence_structures.py  # 1 doctest failed
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/finance/stock.py  # 1 doctest failed
----------------------------------------------------------------------
Features detected for doctesting: dvipng,ffmpeg,graphviz,imagemagick,internet,latex,lualatex,octave,pandoc,pdf2svg,pdflatex,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,xelatex

Therefore, there are no new failing test compared to what is currently the case with respect to known external failing doctests, see #25536. The difference is that now there is a list of features to be detected (pandoc, pdf2svg, etc.) which are now considered as optional instead of external which is a good thing.

I confirm that the new list of external packages is shorter and made only of the following:

>>> s = "4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,ffmpeg,graphviz,imagemagick,jupymake,kenzo,latex,latte_int,lrslib,lualatex,mcqd,meataxe,pandoc,pdf2svg,pdflatex,plantri,pynormaliz,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,tdlib,xelatex"
>>> t = "4ti2,benzene,bliss,buckygen,conway_polynomials,cplex,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,ffmpeg,graphviz,gurobi,imagemagick,internet,jupymake,kenzo,latex,latte_int,lrslib,lualatex,macaulay2,magma,maple,mathematica,matlab,mcqd,meataxe,octave,pandoc,pdf2svg,pdflatex,plantri,pynormaliz,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,scilab,tdlib,xelatex"
>>> set(t.split(',')) - set(s.split(','))
set(['macaulay2', 'mathematica', 'cplex', 'magma', 'gurobi', 'matlab', 'octave', 'internet', 'scilab', 'maple'])

Answer 63 · 2021-11-18T14:18:11.000Z

Reviewer: Sébastien Labbé

Answer 64 · 2021-11-18T14:18:11.000Z

comment:46

Objectives of the ticket are achieved.

Answer 65 · 2021-11-18T18:07:28.000Z

comment:47

Thank you for reviewing!

Answer 66 · 2021-11-23T23:25:37.000Z

comment:48

sage -t --long --warn-long 45.7 --random-seed=86523171498431921424879269621082549712 src/sage/misc/latex.py
**********************************************************************
File "src/sage/misc/latex.py", line 1549, in sage.misc.latex.Latex.add_package_to_preamble_if_available
Failed example:
    latex.extra_preamble()       # optional - latex
Expected:
    '\\usepackage{xypic}\n'
Got:
    ''
**********************************************************************
File "src/sage/misc/latex.py", line 1859, in sage.misc.latex.?
Failed example:
    _run_latex_(file, engine="pdflatex") # optional - latex
Expected:
    'pdf'
Got:
    An error occurred.
    This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020) (preloaded format=pdflatex 2020.11.22)  24 NOV 2021 00:24
    entering extended mode
     restricted \write18 enabled.
     %&-line parsing enabled.
    **\nonstopmode \input{temp.tex}
    (./temp.tex (/usr/share/texlive/texmf-dist/tex/latex/base/article.cls
    Document Class: article 2019/12/20 v1.4l Standard LaTeX document class
    (/usr/share/texlive/texmf-dist/tex/latex/base/size10.clo
    File: size10.clo 2019/12/20 v1.4l Standard LaTeX file (size option)
    )
    \c@part=\count167
    \c@section=\count168
    \c@subsection=\count169
    \c@subsubsection=\count170
    \c@paragraph=\count171
    \c@subparagraph=\count172
    \c@figure=\count173
    \c@table=\count174
    \abovecaptionskip=\skip47
    \belowcaptionskip=\skip48
    \bibindent=\dimen134
    )
    (/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsmath.sty
    Package: amsmath 2020/01/20 v2.17e AMS math features
    \@mathmargin=\skip49
    <BLANKLINE>
    For additional information on amsmath, use the `?' option.
    (/usr/share/texlive/texmf-dist/tex/latex/amsmath/amstext.sty
    Package: amstext 2000/06/29 v2.01 AMS text
    <BLANKLINE>
    (/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsgen.sty
    File: amsgen.sty 1999/11/30 v2.0 generic functions
    \@emptytoks=\toks15
    \ex@=\dimen135
    ))
    (/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsbsy.sty
    Package: amsbsy 1999/11/29 v1.2d Bold Symbols
    \pmbraise@=\dimen136
    )
    (/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsopn.sty
    Package: amsopn 2016/03/08 v2.02 operator names
    )
    \inf@bad=\count175
    LaTeX Info: Redefining \frac on input line 227.
    \uproot@=\count176
    \leftroot@=\count177
    LaTeX Info: Redefining \overline on input line 389.
    \classnum@=\count178
    \DOTSCASE@=\count179
    LaTeX Info: Redefining \ldots on input line 486.
    LaTeX Info: Redefining \dots on input line 489.
    LaTeX Info: Redefining \cdots on input line 610.
    \Mathstrutbox@=\box45
    \strutbox@=\box46
    \big@size=\dimen137
    LaTeX Font Info:    Redeclaring font encoding OML on input line 733.
    LaTeX Font Info:    Redeclaring font encoding OMS on input line 734.
    \macc@depth=\count180
    \c@MaxMatrixCols=\count181
    \dotsspace@=\muskip16
    \c@parentequation=\count182
    \dspbrk@lvl=\count183
    \tag@help=\toks16
    \row@=\count184
    \column@=\count185
    \maxfields@=\count186
    \andhelp@=\toks17
    \eqnshift@=\dimen138
    \alignsep@=\dimen139
    \tagshift@=\dimen140
    \tagwidth@=\dimen141
    \totwidth@=\dimen142
    \lineht@=\dimen143
    \@envbody=\toks18
    \multlinegap=\skip50
    \multlinetaggap=\skip51
    \mathdisplay@stack=\toks19
    LaTeX Info: Redefining \[ on input line 2859.
    LaTeX Info: Redefining \] on input line 2860.
    )
    (/usr/share/texlive/texmf-dist/tex/latex/amsfonts/amssymb.sty
    Package: amssymb 2013/01/14 v3.01 AMS font symbols
    <BLANKLINE>
    (/usr/share/texlive/texmf-dist/tex/latex/amsfonts/amsfonts.sty
    Package: amsfonts 2013/01/14 v3.01 Basic AMSFonts support
    \symAMSa=\mathgroup4
    \symAMSb=\mathgroup5
    LaTeX Font Info:    Redeclaring math symbol \hbar on input line 98.
    LaTeX Font Info:    Overwriting math alphabet `\mathfrak' in version `bold'
    (Font)                  U/euf/m/n --> U/euf/b/n on input line 106.
    ))
    (/usr/share/texlive/texmf-dist/tex/latex/graphics/graphicx.sty
    Package: graphicx 2019/11/30 v1.2a Enhanced LaTeX Graphics (DPC,SPQR)
    <BLANKLINE>
    (/usr/share/texlive/texmf-dist/tex/latex/graphics/keyval.sty
    Package: keyval 2014/10/28 v1.15 key=value parser (DPC)
    \KV@toks@=\toks20
    )
    (/usr/share/texlive/texmf-dist/tex/latex/graphics/graphics.sty
    Package: graphics 2019/11/30 v1.4a Standard LaTeX Graphics (DPC,SPQR)
    <BLANKLINE>
    (/usr/share/texlive/texmf-dist/tex/latex/graphics/trig.sty
    Package: trig 2016/01/03 v1.10 sin cos tan (DPC)
    )
    (/usr/share/texlive/texmf-dist/tex/latex/graphics-cfg/graphics.cfg
    File: graphics.cfg 2016/06/04 v1.11 sample graphics configuration
    )
    Package graphics Info: Driver file: pdftex.def on input line 105.
    <BLANKLINE>
    (/usr/share/texlive/texmf-dist/tex/latex/graphics-def/pdftex.def
    File: pdftex.def 2018/01/08 v1.0l Graphics/color driver for pdftex
    ))
    \Gin@req@height=\dimen144
    \Gin@req@width=\dimen145
    )
    (/usr/share/texlive/texmf-dist/tex/latex/jknapltx/mathrsfs.sty
    Package: mathrsfs 1996/01/01 Math RSFS package v1.0 (jk)
    \symrsfs=\mathgroup6
    )
    (/usr/share/texlive/texmf-dist/tex/latex/base/inputenc.sty
    Package: inputenc 2018/08/11 v1.3c Input encoding file
    \inpenc@prehook=\toks21
    \inpenc@posthook=\toks22
    )
    (/usr/share/texlive/texmf-dist/tex/latex/base/fontenc.sty
    Package: fontenc 2020/02/11 v2.0o Standard LaTeX package
    )
    <BLANKLINE>
    ! LaTeX Error: File `tkz-graph.sty' not found.
    <BLANKLINE>
    Type X to quit or <RETURN> to proceed,
    or enter new name. (Default extension: sty)
    <BLANKLINE>
    Enter file name: 
    ! Emergency stop.
    <read *> 
    <BLANKLINE>
    l.19 \newcommand
                    {\ZZ}{\Bold{Z}}^^M
    *** (cannot \read from terminal in nonstop modes)
    <BLANKLINE>
    <BLANKLINE>
    Here is how much of TeX's memory you used:
     1949 strings out of 482029
     21727 string characters out of 5943690
     251133 words of memory out of 5000000
     17640 multiletter control sequences out of 15000+600000
     533101 words of font info for 25 fonts, out of 8000000 for 9000
     430 hyphenation exceptions out of 8191
     32i,0n,33p,241b,38s stack positions out of 5000i,500n,10000p,200000b,80000s
    !  ==> Fatal error occurred, no output PDF file produced!
    <BLANKLINE>
    'Error latexing slide.'
**********************************************************************
2 items had failures:
   1 of  19 in sage.misc.latex.?
   1 of   5 in sage.misc.latex.Latex.add_package_to_preamble_if_available
    [258 tests, 2 failures, 0.85 s]
----------------------------------------------------------------------
sage -t --long --warn-long 45.7 --random-seed=86523171498431921424879269621082549712 src/sage/misc/latex.py  # 2 doctests failed
----------------------------------------------------------------------
Total time for all tests: 1.0 seconds
    cpu time: 0.2 seconds
    cumulative wall time: 0.9 seconds
Features detected for doctesting: imagemagick,latex

Answer 67 · 2021-11-24T01:59:42.000Z

Changed commit from 12a2507 to 07b2512

Answer 68 · 2021-11-24T01:59:42.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`0dd76b6`	`Merge tag '9.5.beta7' into t/32174/doctests__detect__safe__external_features_even_if____optional_external__is_not_used`
`07b2512`	`src/sage/misc/latex.py: Conditionalize some doctests on latex_package_tkz_graph`

Answer 69 · 2021-11-24T02:00:45.000Z

comment:50

Here's a quick fix by marking some doctests # optional - latex_package_tkz_graph.
For now, this annotation just disables that line. This feature is introduced in #32926.

Answer 70 · 2021-11-25T10:42:00.000Z

comment:52

I do not have access to a computer with no installation of latex. But, I confirm that this

sage -tp --optional=sage,optional,external src/sage/misc/latex.py

works on my machine:

Git branch: 32174
Using --optional=4ti2,ccache,cryptominisat,debian,dot2tex,e_antic,external,fricas,glucose,latte_int,lidia,normaliz,notedown,pandoc_attributes,pip,pycosat,pynormaliz,rst2ipynb,sage,sage_numerical_backends_coin,sage_spkg
Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,cplex,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,ffmpeg,graphviz,gurobi,imagemagick,internet,jupymake,kenzo,latex,latte_int,lrslib,lualatex,macaulay2,magma,maple,mathematica,matlab,mcqd,meataxe,octave,pandoc,pdf2svg,pdflatex,plantri,pynormaliz,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,scilab,tdlib,xelatex
Doctesting 1 file using 8 threads.
sage -t --random-seed=278462588081941083872300580472279407754 src/sage/misc/latex.py
    [255 tests, 1.15 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Features detected for doctesting: imagemagick,latex

Answer 71 · 2021-11-25T10:47:36.000Z

comment:53

All tests are passing on the patchbots. For me it is a positive review.

Can someone having a machine similar to the one used by Volker confirm it is okay?

Volker, are you okay to test it again on your machine?

Answer 72 · 2021-11-26T03:34:35.000Z

comment:54

Let's just put it back in the queue

Answer 73 · 2021-12-01T10:57:22.000Z

comment:55

The 89 + 63 doctest failures (seen on patchbot labri-vindelecroix-1) that are discussed in #32926 are also seen here on the same patchbot testing this ticket #32174. Moreover those numerous failures are not seen by the same patchbot while testing #32866 on which #32174 is based on. So I switch the status of the current ticket to needs info.

Answer 74 · 2021-12-01T11:00:55.000Z

comment:56

(I copy paste here a comment originally written at #32926 comment:20)

All of the 89 + 63 errors are the same. It is always a warning which is printed (the output is always ok) saying ! LaTeX Error: File `type1ec.sty' not found.

Also the machine on which the failures is happening has the following detected features:

Features detected for doctesting: 4ti2,dvipng,ffmpeg,graphviz,latex,latex_package_tkz_graph,lrslib,lualatex,pdflatex,
sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,
sage.rings.real_double,sage.symbolic,xelatex

Here is one of the failure:

**********************************************************************
File "src/sage/plot/plot.py", line 1919, in sage.plot.plot.plot
Failed example:
    parametric_plot((sqrt(x^2-1),sqrt(x^2-1/2)), (x,0,5), exclude=[1,2,3])
Expected:
    Graphics object consisting of 3 graphics primitives
Got:
    doctest:warning
      File "/home/debian/sage/src/bin/sage-runtests", line 155, in <module>
        err = DC.run()
      File "/home/debian/sage/local/lib/python3.9/site-packages/sage/doctest/control.py", line 1253, in run
        self.run_doctests()
      File "/home/debian/sage/local/lib/python3.9/site-packages/sage/doctest/control.py", line 954, in run_doctests
        self.dispatcher.dispatch()
      File "/home/debian/sage/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 1996, in dispatch
        self.parallel_dispatch()
      File "/home/debian/sage/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 1891, in parallel_dispatch
        w.start()  # This might take some time
      File "/home/debian/sage/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 2165, in start
        super(DocTestWorker, self).start()
      File "/usr/lib/python3.9/multiprocessing/process.py", line 121, in start
        self._popen = self._Popen(self)
      File "/usr/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
        return _default_context.get_context().Process._Popen(process_obj)
      File "/usr/lib/python3.9/multiprocessing/context.py", line 277, in _Popen
        return Popen(process_obj)
      File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
        self._launch(process_obj)
      File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 71, in _launch
        code = process_obj._bootstrap(parent_sentinel=child_r)
      File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
        self.run()
      File "/home/debian/sage/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 2137, in run
        task(self.options, self.outtmpfile, msgpipe, self.result_queue)
      File "/home/debian/sage/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 2467, in __call__
        doctests, extras = self._run(runner, options, results)
      File "/home/debian/sage/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 2519, in _run
        result = runner.run(test)
      File "/home/debian/sage/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 861, in run
        return self._run(test, compileflags, out)
      File "/home/debian/sage/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/debian/sage/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.plot.plot.plot[144]>", line 1, in <module>
        parametric_plot((sqrt(x**Integer(2)-Integer(1)),sqrt(x**Integer(2)-Integer(1)/Integer(2))), (x,Integer(0),Integer(5)), exclude=[Integer(1),Integer(2),Integer(3)])
      File "/home/debian/sage/local/lib/python3.9/site-packages/sage/repl/rich_output/display_manager.py", line 825, in displayhook
        plain_text, rich_output = self._rich_output_formatter(obj, dict())
      File "/home/debian/sage/local/lib/python3.9/site-packages/sage/repl/rich_output/display_manager.py", line 643, in _rich_output_formatter
        rich_output = self._call_rich_repr(obj, rich_repr_kwds)
      File "/home/debian/sage/local/lib/python3.9/site-packages/sage/repl/rich_output/display_manager.py", line 608, in _call_rich_repr
        warnings.warn(
      File "/usr/lib/python3.9/warnings.py", line 109, in _showwarnmsg
        sw(msg.message, msg.category, msg.filename, msg.lineno,
    :
    sage.repl.rich_output.display_manager.RichReprWarning: Exception in _rich_repr_ while displaying object: latex was not able to process the following string:
    b'lp'
    <BLANKLINE>
    Here is the full report generated by latex:
    This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020/Debian) (preloaded format=latex)
     restricted \write18 enabled.
    entering extended mode
    <BLANKLINE>
    (/home/debian/.sage/matplotlib-1.5.1/tex.cache/1acea6f6c115d0ec7a634ed0529287b9
    .tex
    LaTeX2e <2020-10-01> patch level 4
    L3 programming layer <2021-01-09> xparse <2020-03-03>
    (/usr/share/texlive/texmf-dist/tex/latex/base/article.cls
    Document Class: article 2020/04/10 v1.4m Standard LaTeX document class
    (/usr/share/texlive/texmf-dist/tex/latex/base/size10.clo))
    (/usr/share/texlive/texmf-dist/tex/latex/type1cm/type1cm.sty)
    <BLANKLINE>
    ! LaTeX Error: File `type1ec.sty' not found.
    <BLANKLINE>
    Type X to quit or <RETURN> to proceed,
    or enter new name. (Default extension: sty)
    <BLANKLINE>
    Enter file name: 
    ! Emergency stop.
    <read *> 
    <BLANKLINE>
    l.6 \usepackage
                   {type1ec}^^M
    No pages of output.
    Transcript written on 1acea6f6c115d0ec7a634ed0529287b9.log.
    <BLANKLINE>
    <BLANKLINE>
    <BLANKLINE>
    Graphics object consisting of 3 graphics primitives

I am not able to reproduce these failures on my machine. I will try to see if these failures appear with the same machine running doctest on different tickets or on no ticket at all (if I remember well, there is a way to check this no?)

Answer 75 · 2021-12-01T11:19:24.000Z

comment:57

See similar discussion in a matplotlib issue:

matplotlib/matplotlib#16911

The fix suggested there is:

$ apt install cm-super

Answer 76 · 2021-12-01T11:21:18.000Z

Description changed:

--- 
+++ 
@@ -6,3 +6,56 @@
 
 This is a step toward #30746 (`sage.doctest.control`: Replace use of `sage.misc.package.list_packages`)
 
+With this ticket, more doctests will be run by default. This may lead to new failures (discovered by patchbot for instance). But that does not mean it is introduced in this ticket, just that they were existing before this ticket, but simply not executed.
+
+For example, let's compare the behavior of the following command:
+
+```
+$ sage -tp --long --show-skipped src/sage/plot/plot.py src/sage/plot/graphics.py
+```
+
+returns:
+
+BEFORE (9.5.beta7):
+
+```
+Git branch: develop
+Using --optional=4ti2,build,ccache,cryptominisat,debian,dochtml,dot2tex,e_antic,fricas,glucose,latte_int,lidia,normaliz,notedown,pandoc_attributes,pip,pycosat,pynormaliz,rst2ipynb,sage,sage.geometry.polyhedron,sage.rings.real_double,sage_numerical_backends_coin,sage_spkg
+Doctesting 2 files using 8 threads.
+sage -t --long --random-seed=120175723758312766297483571135112496058 src/sage/plot/graphics.py
+    2 latex tests not run
+    3 not tested tests not run
+    0 tests not run because we ran out of time
+    [405 tests, 33.39 s]
+sage -t --long --random-seed=120175723758312766297483571135112496058 src/sage/plot/plot.py
+    1 latex test not run
+    0 tests not run because we ran out of time
+    [461 tests, 57.38 s]
+----------------------------------------------------------------------
+All tests passed!
+----------------------------------------------------------------------
+```
+
+AFTER (9.5.beta7 + #32174):
+
+```
+Git branch: 32174
+Using --optional=4ti2,build,ccache,cryptominisat,debian,dochtml,dot2tex,e_antic,fricas,glucose,latte_int,lidia,normaliz,notedown,pandoc_attributes,pip,pycosat,pynormaliz,rst2ipynb,sage,sage_numerical_backends_coin,sage_spkg
+Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,ffmpeg,graphviz,imagemagick,jupymake,kenzo,latex,latte_int,lrslib,lualatex,mcqd,meataxe,pandoc,pdf2svg,pdflatex,plantri,pynormaliz,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,tdlib,xelatex
+Doctesting 2 files using 8 threads.
+sage -t --long --random-seed=67779290317494917173046340853116306275 src/sage/plot/graphics.py
+    2 latex tests not run
+    3 not tested tests not run
+    0 tests not run because we ran out of time
+    [407 tests, 34.35 s]
+sage -t --long --random-seed=67779290317494917173046340853116306275 src/sage/plot/plot.py
+    1 latex test not run
+    0 tests not run because we ran out of time
+    [462 tests, 57.75 s]
+----------------------------------------------------------------------
+All tests passed!
+----------------------------------------------------------------------
+Features detected for doctesting: latex
+```
+
+We observe that the number of tests run went from 405 to 407 and from 461 to 462, because the feature latex was detected.

Answer 77 · 2021-12-01T11:21:18.000Z

comment:58

I updated the description of the ticket to mention that more doctests will be run by default after #32174 which may lead to "new" bugs that were there before. That being said, I still don't know if the 89 + 63 fall in this category, because only 2 + 1 doctests are marked as optional - latex in those two files.

Also, it made me realize that there is a bug with --show-skipped which does not work correctly:
The fact that 2 latex tests not run, 1 latex test not run still appear is a bug, since those tests are run.

Answer 78 · 2021-12-01T11:39:28.000Z

Description changed:

--- 
+++ 
@@ -58,4 +58,4 @@
 Features detected for doctesting: latex
 ```
 
-We observe that the number of tests run went from 405 to 407 and from 461 to 462, because the feature latex was detected.
+We observe that the number of tests run went from 405 to 407 and from 461 to 462, because the feature latex was detected. The fact that `1 latex test not run` still appear is a bug.

Answer 79 · 2021-12-01T11:50:03.000Z

comment:60

The 89 + 63 failures seems to be created by the current branch when looking at the patchbot report made by vindelecroix-labri1 machine. But, when looking at the code of this ticket, I can't see anything which can explain those failures to appear. I am adding machine's owner in cc in case Vincent has an idea.

Most of the machines are fine, so I am fine with putting the status back to positive review.

Or do we want to mark those doctests as dependent on the external latex feature cm-super? That would rather be done in #32926.

Answer 80 · 2021-12-01T13:17:21.000Z

comment:61

Replying to @seblabbe:

Also, it made me realize that there is a bug with --show-skipped which does not work correctly:
The fact that 2 latex tests not run, 1 latex test not run still appear is a bug, since those tests are run.

I am adding a commit to fix the issue with the --show-skipped report.

New commits:

`cb13437`	`32174: fix the --show-skipped doctest report`

Answer 81 · 2021-12-01T13:17:21.000Z

Changed commit from 07b2512 to cb13437

Answer 82 · 2021-12-01T13:17:21.000Z

Changed branch from u/mkoeppe/doctests__detect__safe__external_features_even_if____optional_external__is_not_used to public/32174

Answer 83 · 2021-12-01T13:18:52.000Z

Description changed:

--- 
+++ 
@@ -44,12 +44,10 @@
 Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,ffmpeg,graphviz,imagemagick,jupymake,kenzo,latex,latte_int,lrslib,lualatex,mcqd,meataxe,pandoc,pdf2svg,pdflatex,plantri,pynormaliz,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,tdlib,xelatex
 Doctesting 2 files using 8 threads.
 sage -t --long --random-seed=67779290317494917173046340853116306275 src/sage/plot/graphics.py
-    2 latex tests not run
     3 not tested tests not run
     0 tests not run because we ran out of time
     [407 tests, 34.35 s]
 sage -t --long --random-seed=67779290317494917173046340853116306275 src/sage/plot/plot.py
-    1 latex test not run
     0 tests not run because we ran out of time
     [462 tests, 57.75 s]
 ----------------------------------------------------------------------
@@ -58,4 +56,4 @@
 Features detected for doctesting: latex
 ```
 
-We observe that the number of tests run went from 405 to 407 and from 461 to 462, because the feature latex was detected. The fact that `1 latex test not run` still appear is a bug.
+We observe that the number of tests run went from 405 to 407 and from 461 to 462, because the feature latex was detected.

Answer 84 · 2021-12-01T17:23:26.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`391dca1`	`sage.misc.latex.png: Mark doctest # optional - imagemagick`

Answer 85 · 2021-12-01T17:23:26.000Z

Changed commit from cb13437 to 391dca1

Answer 86 · 2021-12-01T17:23:42.000Z

comment:64

Cherry-picked from #32926.

Answer 87 · 2021-12-01T17:26:38.000Z

Changed commit from 391dca1 to 6f311b3

Answer 88 · 2021-12-01T17:26:38.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`6f311b3`	`src/sage/doctest/external.py: Revert latex features to 'external'`

Answer 89 · 2021-12-01T17:29:03.000Z

comment:66

In order to get this ticket in, I have now reduced the scope, keeping the latex features as external. In a follow-up ticket, we can work on improving the # optional tags.

Answer 90 · 2021-12-01T17:55:04.000Z

comment:67

updated the example in the description of the ticket to be based on pandoc instead of latex

Answer 91 · 2021-12-01T17:55:04.000Z

Description changed:

--- 
+++ 
@@ -11,7 +11,7 @@
 For example, let's compare the behavior of the following command:
 
 ```
-$ sage -tp --long --show-skipped src/sage/plot/plot.py src/sage/plot/graphics.py
+sage -tp --show-skipped src/sage/tests/cmdline.py 
 ```
 
 returns:
@@ -21,16 +21,14 @@
 ```
 Git branch: develop
 Using --optional=4ti2,build,ccache,cryptominisat,debian,dochtml,dot2tex,e_antic,fricas,glucose,latte_int,lidia,normaliz,notedown,pandoc_attributes,pip,pycosat,pynormaliz,rst2ipynb,sage,sage.geometry.polyhedron,sage.rings.real_double,sage_numerical_backends_coin,sage_spkg
-Doctesting 2 files using 8 threads.
-sage -t --long --random-seed=120175723758312766297483571135112496058 src/sage/plot/graphics.py
-    2 latex tests not run
-    3 not tested tests not run
+Doctesting 1 file using 8 threads.
+sage -t --random-seed=265206128165572261397002909756599539048 src/sage/tests/cmdline.py
+    3 gdb tests not run
+    12 internet tests not run
+    2 pandoc tests not run
+    4 r tests not run
     0 tests not run because we ran out of time
-    [405 tests, 33.39 s]
-sage -t --long --random-seed=120175723758312766297483571135112496058 src/sage/plot/plot.py
-    1 latex test not run
-    0 tests not run because we ran out of time
-    [461 tests, 57.38 s]
+    [235 tests, 30.65 s]
 ----------------------------------------------------------------------
 All tests passed!
 ----------------------------------------------------------------------
@@ -39,21 +37,32 @@
 AFTER (9.5.beta7 + #32174):
 
 ```
+$ sage -tp --show-skipped src/sage/tests/cmdline.py 
 Git branch: 32174
 Using --optional=4ti2,build,ccache,cryptominisat,debian,dochtml,dot2tex,e_antic,fricas,glucose,latte_int,lidia,normaliz,notedown,pandoc_attributes,pip,pycosat,pynormaliz,rst2ipynb,sage,sage_numerical_backends_coin,sage_spkg
-Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,ffmpeg,graphviz,imagemagick,jupymake,kenzo,latex,latte_int,lrslib,lualatex,mcqd,meataxe,pandoc,pdf2svg,pdflatex,plantri,pynormaliz,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,tdlib,xelatex
-Doctesting 2 files using 8 threads.
-sage -t --long --random-seed=67779290317494917173046340853116306275 src/sage/plot/graphics.py
-    3 not tested tests not run
+Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,ffmpeg,graphviz,imagemagick,jupymake,kenzo,latte_int,lrslib,mcqd,meataxe,pandoc,pdf2svg,plantri,pynormaliz,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,tdlib
+Doctesting 1 file using 8 threads.
+sage -t --random-seed=100100306103974678574881156032164018710 src/sage/tests/cmdline.py
+**********************************************************************
+File "src/sage/tests/cmdline.py", line 732, in sage.tests.cmdline.test_executable
+Failed example:
+    print(open(output, 'r').read() == t)          # optional - pandoc
+Expected:
+    True
+Got:
+    False
+**********************************************************************
+1 item had failures:
+   1 of 238 in sage.tests.cmdline.test_executable
+    3 gdb tests not run
+    12 internet tests not run
+    4 r tests not run
     0 tests not run because we ran out of time
-    [407 tests, 34.35 s]
-sage -t --long --random-seed=67779290317494917173046340853116306275 src/sage/plot/plot.py
-    0 tests not run because we ran out of time
-    [462 tests, 57.75 s]
+    [237 tests, 1 failure, 32.76 s]
 ----------------------------------------------------------------------
-All tests passed!
+sage -t --random-seed=100100306103974678574881156032164018710 src/sage/tests/cmdline.py  # 1 doctest failed
 ----------------------------------------------------------------------
-Features detected for doctesting: latex
+Features detected for doctesting: pandoc
 ```
 
-We observe that the number of tests run went from 405 to 407 and from 461 to 462, because the feature latex was detected.
+We observe that the number of tests run went from 235 to 237, because the feature pandoc was detected. Note that the doctest failure is already tracked at #32697.

Answer 92 · 2021-12-01T17:58:51.000Z

comment:68

Replying to @mkoeppe:

In a follow-up ticket, we can work on improving the # optional tags.

This is now #32955.

Answer 93 · 2021-12-01T18:02:45.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`3bfe1a1`	`32174: adapting the doctest listing external softwares`

Answer 94 · 2021-12-01T18:02:45.000Z

Changed commit from 6f311b3 to 3bfe1a1

Answer 95 · 2021-12-01T18:06:12.000Z

Changed author from Matthias Koeppe to Matthias Koeppe, Sébastien Labbé

Answer 96 · 2021-12-01T18:11:32.000Z

comment:71

I started a re-run of all tests on my machine. Will report later.

@Matthias: Sorry for having re-open this ticket earlier today. It was having a positive review for 7 days. I should have open another ticket instead I think. I hope it will not delay the entry of this ticket into sage since many other tickets depend on it. I will stay close to it to not add anymore delay.

Answer 97 · 2021-12-01T18:14:29.000Z

comment:72

No worries, and thanks for all your work!

Answer 98 · 2021-12-01T19:40:45.000Z

comment:73

Replying to @seblabbe:

I started a re-run of all tests on my machine. Will report later.

On my machine running optional and external doctests on top of 9.5.beta7 returns:

----------------------------------------------------------------------
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/graphs/generic_graph.py  # 2 doctests failed
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/databases/findstat.py  # 9 doctests failed
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/tests/cmdline.py  # 10 doctests failed
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/graphs/graph.py  # 3 doctests failed
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/interfaces/fricas.py  # 1 doctest failed
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/sat/solvers/dimacs.py  # 2 doctests failed
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/combinat/designs/incidence_structures.py  # 1 doctest failed
sage -t --long --random-seed=199830843512308713498063069261640048262 src/sage/finance/stock.py  # 1 doctest failed
----------------------------------------------------------------------
Total time for all tests: 2664.6 seconds
    cpu time: 16553.0 seconds
    cumulative wall time: 20701.4 seconds
Features detected for doctesting: dvipng,ffmpeg,graphviz,imagemagick,internet,latex,lualatex,octave,pandoc,pdf2svg,pdflatex,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,xelatex

which has no more failures than what I reported for 9.5.beta7 on sage-release.

I am waiting for the patchbot to finish.

Answer 99 · 2021-12-02T09:37:03.000Z

comment:74

The patchbot panke finished its tests. It has only one failure which is exactly the one given in the description of this ticket which I repeat is already tracked at #32697:

sage -t --long --random-seed=318717441882109214278396779393985383988 src/sage/tests/cmdline.py
**********************************************************************
File "src/sage/tests/cmdline.py", line 732, in sage.tests.cmdline.test_executable
Failed example:
    print(open(output, 'r').read() == t)          # optional - pandoc
Expected:
    True
Got:
    False
**********************************************************************
1 item had failures:
   1 of 223 in sage.tests.cmdline.test_executable
    [222 tests, 1 failure, 57.81 s]

From me it is a positive review. Next stuff to be checked are

whether it works on vindelecroix-labri1 machine (are the 63 + 89 failures gone with the last changes by Matthias? personnaly, I am not even sure). That being said Vincent is gone for vacation for 4 weeks, so we might not want to wait for an answer.
whether it works on Volker's machine (normally this was fixed by 07b2512)

Answer 100 · 2021-12-02T18:10:53.000Z

comment:75

Replying to @seblabbe:

whether it works on vindelecroix-labri1 machine (are the 63 + 89 failures gone with the last changes by Matthias?

Well, these tests are # optional - latex, and latex is now no longer automatically added unless external is in use.