sage.doctest.control: Replace use of sage.misc.package.list_packages
mkoeppe opened this issue · 28 comments
... by features.
This is so that users can make use of doctests conditionalized with "# optional" in their Sage programs even on distribution-packaged Sage installations, or for optional packages with spkg-configure.m4 (as observed in #30887 comment:28)
After #30887, #32614, #32649 a Feature has a name attribute that can be used as the # optional tag for doctesting.
#32174 implements global Feature discovery and hooks it into sage.doctest.control.
It remains to define Features for all remaining SPKGs that appear in # optional tags somewhere in the Sage library (which can be found using git grep -l '# *optional' | xargs sed -E -n 's/^.*# *optional[ -:]*([a-zA-Z_0-9.]*).*/\1/gp' | sort -u). Then we can remove the use of sage.misc.package.
Related:
-
#30848
sage.doctest.control,sage_setup: Do not check versions of installed packages -
Depends on: #35856
CC: @antonio-rojas @kiwifb @simon-king-jena @dimpase @jhpalmieri @kliem @tscrim @kwankyu @orlitzky
Component: doctest framework
Keywords: sd111
Issue created by migration from https://trac.sagemath.org/ticket/30746
How do distributions handle this currently?
Description changed:
---
+++
@@ -2,3 +2,4 @@
This is so that users can make use of doctests conditionalized with "# optional" in their Sage programs even on distribution-packaged Sage installations.
+This is also preparation for #30747 (deprecate all of `sage.misc.package`)I'm patching it out completely
https://github.com/archlinux/svntogit-community/blob/packages/sagemath/trunk/test-optional.patch
so tests depending on optional packages are skipped unless explicitely specified in the sage -t command
Replying to @mkoeppe:
How do distributions handle this currently?
I don't know about others but I do it badly. The current stuff allow some autodetection of what is installed by the sage package management system. That was not always the case. I remove the autodetection bits and add anything I want to test manually when running sage -t.
Description changed:
---
+++
@@ -1,4 +1,4 @@
-... by features.
+... by features or `sage_conf`.
This is so that users can make use of doctests conditionalized with "# optional" in their Sage programs even on distribution-packaged Sage installations.
(wrong ticket)
Changed branch from u/mkoeppe/sage_doctest_control__replace_use_of_sage_misc_package_list_packages to none
Changed keywords from none to sd111
Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111
Setting new milestone based on a cursory review of ticket status, priority, and last modification date.
Description changed:
---
+++
@@ -1,5 +1,7 @@
... by features or `sage_conf`.
-This is so that users can make use of doctests conditionalized with "# optional" in their Sage programs even on distribution-packaged Sage installations.
+This is so that users can make use of doctests conditionalized with "# optional" in their Sage programs even on distribution-packaged Sage installations, or for optional packages with `spkg-configure.m4` (as observed in [#30887 comment:28](https://github.com/sagemath/sage/issues/30887#comment:28))
This is also preparation for #30747 (deprecate all of `sage.misc.package`)
+
+Related: #30848 `sage.doctest.control`, `sage_setup`: Do not check versions of installed packagesWhy the dependency on #20879? I am assuming this is a typo or the wrong ticket.
Description changed:
---
+++
@@ -1,6 +1,8 @@
-... by features or `sage_conf`.
+... by features.
This is so that users can make use of doctests conditionalized with "# optional" in their Sage programs even on distribution-packaged Sage installations, or for optional packages with `spkg-configure.m4` (as observed in [#30887 comment:28](https://github.com/sagemath/sage/issues/30887#comment:28))
+
+The new features added in #30887, #32614 already have a `name` attribute that can be used as the "optional" tag for doctesting. It just needs to be hooked into `sage.doctest.control` - in the same way that #32614's `sage_features` is done.
This is also preparation for #30747 (deprecate all of `sage.misc.package`)
Description changed:
---
+++
@@ -6,4 +6,6 @@
This is also preparation for #30747 (deprecate all of `sage.misc.package`)
-Related: #30848 `sage.doctest.control`, `sage_setup`: Do not check versions of installed packages
+Related:
+- #30848 `sage.doctest.control`, `sage_setup`: Do not check versions of installed packages
+- #32174 doctests: Detect "safe" external features even if "--optional=external" is not usedDescription changed:
---
+++
@@ -2,7 +2,7 @@
This is so that users can make use of doctests conditionalized with "# optional" in their Sage programs even on distribution-packaged Sage installations, or for optional packages with `spkg-configure.m4` (as observed in [#30887 comment:28](https://github.com/sagemath/sage/issues/30887#comment:28))
-The new features added in #30887, #32614 already have a `name` attribute that can be used as the "optional" tag for doctesting. It just needs to be hooked into `sage.doctest.control` - in the same way that #32614's `sage_features` is done.
+After #30887, #32614, #32649 a `Feature` has a `name` attribute that can be used as the "optional" tag for doctesting. It just needs to be hooked into `sage.doctest.control` - in the same way that #32614's `sage_features` is done.
This is also preparation for #30747 (deprecate all of `sage.misc.package`)
Description changed:
---
+++
@@ -2,10 +2,14 @@
This is so that users can make use of doctests conditionalized with "# optional" in their Sage programs even on distribution-packaged Sage installations, or for optional packages with `spkg-configure.m4` (as observed in [#30887 comment:28](https://github.com/sagemath/sage/issues/30887#comment:28))
-After #30887, #32614, #32649 a `Feature` has a `name` attribute that can be used as the "optional" tag for doctesting. It just needs to be hooked into `sage.doctest.control` - in the same way that #32614's `sage_features` is done.
+After #30887, #32614, #32649 a `Feature` has a `name` attribute that can be used as the `# optional` tag for doctesting.
+
+#32174 implements global `Feature` discovery and hooks it into `sage.doctest.control`.
+
+It remains to define `Feature`s for all remaining SPKGs that appear in `# optional` tags somewhere in the Sage library (which can be found using `git grep -l '# *optional' | xargs sed -E -n 's/^.*# *optional[ -:]*([a-zA-Z_0-9.]*).*/\1/gp' | sort -u`). Then we can remove the use of `sage.misc.package`.
This is also preparation for #30747 (deprecate all of `sage.misc.package`)
Related:
- #30848 `sage.doctest.control`, `sage_setup`: Do not check versions of installed packages
-- #32174 doctests: Detect "safe" external features even if "--optional=external" is not used
+I'm just seeing this now after wondering why sage keeps searching my system for random broken programs to try to use. Runtime detection of these things will not be a good long-term approach:
- You have to reimplement every spkg-configure.m4 in python for the feature check and keep them synchronized forever.
- Runtime checks are run over and over again. And many are going to be slow, once they are made to return the right answers.
- Distros track dependencies and usually don't want them to (dis)appear at runtime.
- If I build sage with e.g.
--disable-lrslib, I expect it to not search my system for something that looks like lrslib and use it anyway. There has to be a way to reliably disable things.
As an alternative, you can imagine a shared location where "features" record their existence (think pkg-config). The simplest implementation might be to create a file named after the feature to indicate that the feature is present.
Optional packages would touch $FEATURESDIR/$PACKAGE upon being detected or installed. That centralizes all of the detection code in spkg-configure.m4, and would eventually make --disable-foo work again. Distros could install a "feature file" for every feature that their distro package provides. Or if they don't mind the automagic dependencies, you could have (say) the "ffmpeg" package install the ffmpeg feature file to tell sage that ffmpeg is available. Of course this would only be done if the distro's ffmpeg package actually works with its sage. But it does allow the user to add features on a binary distro on-the-fly without the developers having to jump through too many hoops. (On Gentoo we might just make you rebuild with USE=feature to enable it and encode the dependency.)
Depending on the scenario, this essentially runs the "feature check" either once or never, as [ -f $feature ] is free. Thus we avoid wasting time on every invocation of sage -t.
The file-based implementation here is not well-thought-out and probably has flaws, but it's also not what's important. What is is that the features should respect the user/distro configuration, not waste too many resources, not cause surprises, and not require any unnecessary duplication (i.e. work). Moving towards runtime-only detection is IMHO moving backwards.
+1 on a mechanism for statically declaring features present/not present, for use by downstream distributions. They could be provided through sage_conf.
-1 on defining file system stuff, config files, etc.
Replying to @orlitzky:
- You have to reimplement every spkg-configure.m4 in python for the feature check and keep them synchronized forever.
A more positive perspective on that would be that eventually, when we can assume that there is an at least mildly usable python3 available at ./configure time, we can replace many spkg-configure.m4 tests by Feature tests, written in Python.
Replying to @mkoeppe:
+1 on a mechanism for statically declaring features present/not present, for use by downstream distributions. They could be provided through
sage_conf.-1 on defining file system stuff, config files, etc.
With sleep comes clarity. Instead of something like pkg-config, why don't we literally use pkg-config for this? To each feature, associate sage-$feature.pc. This meets all the criteria above without any hacky wheel reinventing. Distros are already familiar with the concept. PKG_CONFIG_PATH tricks should let us avoid version mismatch issues when, say, a git checkout is used on a system where a distro sage is also installed.
A more positive perspective on that would be that eventually, when we can assume that there is an at least mildly usable python3 available at ./configure time, we can replace many spkg-configure.m4 tests by Feature tests, written in Python.
Maybe in some cases, but in general we aren't going to be able to easily replace well-tested autoconf macros with hand-written python. Consequently we'll wind up with the same machinery implemented in two different places in two different languages.
Replying to @orlitzky:
A more positive perspective on that would be that eventually, when we can assume that there is an at least mildly usable python3 available at ./configure time, we can replace many spkg-configure.m4 tests by Feature tests, written in Python.
Maybe in some cases, but in general we aren't going to be able to easily replace well-tested autoconf macros with hand-written python. Consequently we'll wind up with the same machinery implemented in two different places in two different languages.
The trickiest spkg-configure.m4 tests are for compile-time stuff - existence of libraries and such - and they do not need to be implemented in sage.features.