Split sage_setup.docbuild out to a separate distribution sage_docbuild
mkoeppe opened this issue · 80 comments
sage_setup.docbuild, created in #19127, has very different dependencies compared to sage_setup:
It depends on sagelib and sphinx,
whereas the rest of sage_setup is for building sagelib.
We remove the nesting within sage_setup, creating a new top-level package sage_docbuild.
We split it out as a separate pip-installable distribution package sage_docbuild.
Using the spkg-src script, a pip-installable tarball can be generated, which could be uploaded to PyPI -- for the use by external packages that want to build documentation.
This is preparation for #29868 (pip-installable packages sagemath-doc-src, sagemath-doc-inventory, sagemath-doc-html, sagemath-doc-pdf)
Depends on #31344
Depends on #31353
CC: @kiwifb @jhpalmieri @isuruf @mwageringel @haraldschilly @mezzarobba
Component: refactoring
Keywords: sd111
Author: Matthias Koeppe
Branch/Commit: aaab1d3
Reviewer: John Palmieri
Issue created by migration from https://trac.sagemath.org/ticket/30010
Description changed:
---
+++
@@ -3,5 +3,7 @@
It depends on `sagelib` and `sphinx`,
whereas the rest of `sage_setup` is for building `sagelib`.
-We split it out.
+We split it out as a separate distutils package `sage_setup-docbuild`.
+Because some external packages use `sage_setup.docbuild` to build their own documentation, we do not change the package hierarchy. Instead we turn `sage_setup` into a native namespace package.
+Description changed:
---
+++
@@ -3,7 +3,7 @@
It depends on `sagelib` and `sphinx`,
whereas the rest of `sage_setup` is for building `sagelib`.
-We split it out as a separate distutils package `sage_setup-docbuild`.
+We split it out as a separate distutils package `sage_setup.docbuild`.
Because some external packages use `sage_setup.docbuild` to build their own documentation, we do not change the package hierarchy. Instead we turn `sage_setup` into a native namespace package.
Last 10 new commits:
b8e366e | build/pkgs/sagelib/spkg-src: chmod +x |
0473ef3 | Merge branch 't/21559/change-src-bin-installation' into t/29950/build_sagelib_using_installed_sage_setup |
06a3609 | sage_setup.find.find_python_sources: Add benchmark doctest |
174626c | Merge branch 't/29786/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_4__sage_rings_' into t/29701/replace_use_of_module_list_optionalextension |
034a7f7 | Merge branch 'public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' of git://trac.sagemath.org/sage into t/29701/replace_use_of_module_list_optionalextension |
d3c608b | Merge tag '9.2.beta3' into t/29701/replace_use_of_module_list_optionalextension |
7244371 | Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup |
4344f89 | Merge branch 't/21559/change-src-bin-installation' into t/29950/build_sagelib_using_installed_sage_setup |
72c5040 | Merge branch 't/29950/build_sagelib_using_installed_sage_setup' into t/30010/split_sage_setup_docbuild_out_to_a_separate_package |
ce79954 | build/pkgs/sage_setup_docbuild: New |
Author: Matthias Koeppe
Description changed:
---
+++
@@ -3,7 +3,10 @@
It depends on `sagelib` and `sphinx`,
whereas the rest of `sage_setup` is for building `sagelib`.
-We split it out as a separate distutils package `sage_setup.docbuild`.
+We split it out as a separate distutils package `sage_setup.docbuild`.
Because some external packages use `sage_setup.docbuild` to build their own documentation, we do not change the package hierarchy. Instead we turn `sage_setup` into a native namespace package.
+
+Using the `spkg-src` script, a pip-installable tarball can be generated, which could be uploaded to PyPI -- for the use by external packages that want to build documentation.
+Branch pushed to git repo; I updated commit sha1. New commits:
e6b5f75 | build/pkgs/sagelib/src/MANIFEST.in: Prune sage_setup/docbuild |
Branch pushed to git repo; I updated commit sha1. New commits:
cbfe485 | build/pkgs/sage_setup_docbuild: Set version |
In the new spkg-install file, why
. sage-dist-helpers
Why do this at all, and if you're going to do it, why not use source instead of ., to be consistent with how sage-spkg does things?
Branch pushed to git repo; I updated commit sha1. New commits:
39781d1 | build/pkgs/sage_setup_docbuild/spkg-install: Remove useless use of sage-dist-helpers |
Thanks. This was a leftover from when I was using sdh_pip_install.
Can you please add sage_setup_docbuild to DOC_DEPENDENCIES in build/make/Makefile.in?
Edit: No, sorry, it's already there. I just built Sage and it build sage_setup_docbuild at the end, after the documentation. I don't know why.
Hm... it's possible that it is always built because its dependency sagelib is always built...
Branch pushed to git repo; I updated commit sha1. New commits:
fcad518 | Merge tag '9.2.beta7' into t/29701/replace_use_of_module_list_optionalextension |
55c3fbc | src/sage_setup/clean.py: Fix doctest |
01b96b0 | Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup |
8a19fe2 | build/make/Makefile.in (sagelib-clean): Clean the new build location |
ccc67b0 | src/sage_setup: Update cythonized_dir in doctests |
0237faa | Merge branch 't/29950/build_sagelib_using_installed_sage_setup' into t/30010/split_sage_setup_docbuild_out_to_a_separate_package |
Branch pushed to git repo; I updated commit sha1. New commits:
7d1af54 | build/pkgs/sage_setup_docbuild: Make sagelib an order-only dep |
Branch pushed to git repo; I updated commit sha1. New commits:
8aa6fd9 | build/bin/sage-dist-helpers (sdh_pip_install): Build a wheel, store it |
d369aab | build/bin/sage-dist-helpers (sdh_store_and_pip_install_wheel): New, factored out from sdh_pip_install |
2d435ab | build/pkgs/numpy/spkg-install.in: Install via setup.py bdist_wheel |
55993b6 | build/bin/sage-dist-helpers: Fixup |
0a64674 | build/pkgs/gambit/spkg-install.in: Install via bdist_wheel |
ca58693 | build/pkgs/pillow/spkg-install.in: Install via bdist_wheel |
65773f7 | Merge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30010/split_sage_setup_docbuild_out_to_a_separate_package |
14ab446 | build/pkgs/sage_setup_docbuild/spkg-install: Install via setup.py bdist_wheel |
I wrote these comments before the most recent change, but I think they are still relevant:
I propose the following change:
diff --git a/build/pkgs/sage_setup_docbuild/dependencies b/build/pkgs/sage_setup_docbuild/dependencies
index 20f527f0f1..b21990c270 100644
--- a/build/pkgs/sage_setup_docbuild/dependencies
+++ b/build/pkgs/sage_setup_docbuild/dependencies
@@ -1 +1 @@
-$(PYTHON) sphinx six | $(PYTHON_TOOLCHAIN) sagelib
+$(PYTHON) sphinx | $(PYTHON_TOOLCHAIN) sagelib
diff --git a/build/pkgs/sage_setup_docbuild/src/requirements.txt b/build/pkgs/sage_setup_docbuild/src/requirements.txt
index 2dedc630e9..4e5dd84d0e 100644
--- a/build/pkgs/sage_setup_docbuild/src/requirements.txt
+++ b/build/pkgs/sage_setup_docbuild/src/requirements.txt
@@ -1,3 +1,2 @@
#sage
sphinx
-six
diff --git a/src/sage_setup/docbuild/ext/sage_autodoc.py b/src/sage_setup/docbuild/ext/sage_autodoc.py
index 032365d6bd..92d19b883e 100644
--- a/src/sage_setup/docbuild/ext/sage_autodoc.py
+++ b/src/sage_setup/docbuild/ext/sage_autodoc.py
@@ -26,8 +26,6 @@ AUTHORS:
- Kwankyu Lee (2018-12-26): rebase on the latest sphinx.ext.autodoc
"""
-from six import PY2, itervalues, text_type, class_types, string_types
-
import inspect
import re
import sys
@@ -1176,27 +1174,19 @@ class ClassDocumenter(DocstringSignatureMixin, ModuleLevelDocumenter):
if ret:
module = getattr(self.object, '__module__', False)
name = getattr(self.object, '__name__', False)
- if PY2:
- if name and module:
- cls = getattr(sys.modules[module], name, None)
- self.doc_as_attr = (self.objpath != name.split('.') and
- self.object is cls)
- else:
- self.doc_as_attr = True
+ qualname = getattr(self.object, '__qualname__', name)
+ if qualname and module:
+ # walk the standard attribute lookup path for this object
+ qualname_parts = qualname.split('.')
+ cls = getattr(sys.modules[module], qualname_parts[0], None)
+ for part in qualname_parts[1:]:
+ if cls is None:
+ break
+ cls = getattr(cls, part, None)
+ self.doc_as_attr = (self.objpath != qualname_parts and
+ self.object is cls)
else:
- qualname = getattr(self.object, '__qualname__', name)
- if qualname and module:
- # walk the standard attribute lookup path for this object
- qualname_parts = qualname.split('.')
- cls = getattr(sys.modules[module], qualname_parts[0], None)
- for part in qualname_parts[1:]:
- if cls is None:
- break
- cls = getattr(cls, part, None)
- self.doc_as_attr = (self.objpath != qualname_parts and
- self.object is cls)
- else:
- self.doc_as_attr = True
+ self.doc_as_attr = True
return ret
def format_args(self):
-
Also, I don't understand the "tarball" aspects of this. There is no tarball, is there? So why have
spkg-src? -
Is it built in
build/pkgs/sage_setup_docbuild/src/? After building, I see new directoriesbuildandsage_setup.docbuild.egg-infothere. -
The installation log file says
package init file 'sage_setup/__init__.py' not found (or not a regular file). I assume this is not a big issue, since everything worked after installation.
I agree with these changes; could you push them to the branch please?
Replying to @jhpalmieri:
- Also, I don't understand the "tarball" aspects of this. There is no tarball, is there? So why have
spkg-src?
The spkg-src script can build a tarball for the purpose of uploading a source distribution to PyPI.
- Is it built in
build/pkgs/sage_setup_docbuild/src/? After building, I see new directoriesbuildandsage_setup.docbuild.egg-infothere.
That's right.
I should probably add a sage_setup_docbuild-clean target that deletes them.
- The installation log file says
package init file 'sage_setup/__init__.py' not found (or not a regular file). I assume this is not a big issue, since everything worked after installation.
This is not an error. sage_setup is now a native namespace package - these do not have __init__.py files.
Reviewer: John Palmieri
Adding a sage_setup_docbuild-clean would be nice, and/or an spkg-postinst script — can happen either here or on another ticket.
New commits:
4ad7d7e | trac 30010: remove six as a dependency of sage_setup_docbuild |
I'm getting lots of
sage -t --long --warn-long 44.0 --random-seed=0 src/sage_setup/find.py
**********************************************************************
File "src/sage_setup/find.py", line 38, in sage_setup.find.read_distribution
Failed example:
from sage_setup.find import read_distribution
Exception raised:
Traceback (most recent call last):
File "/home/release/Sage/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 720, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/home/release/Sage/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 1145, in compile_and_execute
exec(compiled, globs)
File "<doctest sage_setup.find.read_distribution[1]>", line 1, in <module>
from sage_setup.find import read_distribution
ModuleNotFoundError: No module named 'sage_setup.find'
**********************************************************************
Are incremental builds supported?
I'll revisit this ticket at the start of the 9.3 series
Changed keywords from none to sd111
Description changed:
---
+++
@@ -1,12 +1,12 @@
-`sage_setup.docbuild` has very different dependencies:
+`sage_setup.docbuild` has very different dependencies compared to `sage_setup`:
It depends on `sagelib` and `sphinx`,
whereas the rest of `sage_setup` is for building `sagelib`.
-We split it out as a separate distutils package `sage_setup.docbuild`.
-
-Because some external packages use `sage_setup.docbuild` to build their own documentation, we do not change the package hierarchy. Instead we turn `sage_setup` into a native namespace package.
-
+We split it out as a separate distutils package `sage_docbuild`.
Using the `spkg-src` script, a pip-installable tarball can be generated, which could be uploaded to PyPI -- for the use by external packages that want to build documentation.
+This is preparation for #29868 (pip-installable packages sagemath-doc-src, sagemath-doc-inventory, sagemath-doc-html, sagemath-doc-pdf)
+
+Branch pushed to git repo; I updated commit sha1. New commits:
c76fea5 | src/sage/doctest/control.py: When adding sage_setup, add sage_doctest too |
Branch pushed to git repo; I updated commit sha1. New commits:
c262e44 | build/pkgs/sage_docbuild/src: Update metadata |
Description changed:
---
+++
@@ -3,7 +3,9 @@
It depends on `sagelib` and `sphinx`,
whereas the rest of `sage_setup` is for building `sagelib`.
-We split it out as a separate distutils package `sage_docbuild`.
+We remove the nesting within `sage_setup`, creating a new top-level package `sage_docbuild`.
+
+We split it out as a separate pip-installable distribution package `sage_docbuild`.
Using the `spkg-src` script, a pip-installable tarball can be generated, which could be uploaded to PyPI -- for the use by external packages that want to build documentation.
Description changed:
---
+++
@@ -1,4 +1,4 @@
-`sage_setup.docbuild` has very different dependencies compared to `sage_setup`:
+`sage_setup.docbuild`, created in #19127, has very different dependencies compared to `sage_setup`:
It depends on `sagelib` and `sphinx`,
whereas the rest of `sage_setup` is for building `sagelib`.Doctesting fails for me:
./sage -t -p --all --long --logfile=logs/ptestlong.log
Running doctests with ID 2021-02-23-09-29-26-45521505.
Git branch: t/30010/split_sage_setup_docbuild_out_to_a_separate_package
Using --optional=build,dochtml,homebrew,pip,sage,sage_spkg
Doctesting entire Sage library.
Traceback (most recent call last):
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.beta7/src/bin/sage-runtests", line 182, in <module>
err = DC.run()
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.beta7/local/lib/python3.9/site-packages/sage/doctest/control.py", line 1206, in run
self.expand_files_into_sources()
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.beta7/local/lib/python3.9/site-packages/sage/doctest/control.py", line 790, in expand_files_into_sources
self.sources = [FileDocTestSource(path, self.options) for path in expand()]
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.beta7/local/lib/python3.9/site-packages/sage/doctest/control.py", line 790, in <listcomp>
self.sources = [FileDocTestSource(path, self.options) for path in expand()]
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.beta7/local/lib/python3.9/site-packages/sage/doctest/sources.py", line 527, in __init__
raise ValueError("unknown file extension %r"%ext)
ValueError: unknown file extension ''
make: *** [ptestlong] Error 1
I think the problem is this:
diff --git a/src/sage/doctest/control.py b/src/sage/doctest/control.py
index 7d2b84b9c3..eafcf924de 100644
--- a/src/sage/doctest/control.py
+++ b/src/sage/doctest/control.py
@@ -697,6 +697,7 @@ class DocTestController(SageObject):
# don't make sense to run outside a build environment
if have_git:
self.files.append(opj(SAGE_SRC, 'sage_setup'))
+ self.files.append(opj(SAGE_SRC, 'sage_doctest'))
self.files.append(SAGE_DOC_SRC)
if self.options.all or (self.options.new and not have_git):Should sage_doctest be sage_docbuild instead?
Yes, thanks for catching this! Please push it to the ticket
The directory structure looks very strange: inside SAGE_ROOT/build/pkgs/sage_docbuild/src/ I see
build
lib
build
lib
build
lib
build
lib
sage_docbuild
sage_docbuild
sage_docbuild
sage_docbuild
where indentations indicate levels of subdirectories. For example, I see SAGE_ROOT/build/pkgs/sage_docbuild/src/build/lib/build/lib/build/lib/build/lib/sage_docbuild. All of the sage_docbuild directories look identical.
New commits:
b42920b | trac 30010: (typo) change sage_doctest -> sage_docbuild |
Thanks for catching this! This commit should fix it
New commits:
0a472f2 | build/pkgs/sage_docbuild/src/setup.py: Restrict find_namespace_packages to sage_docbuild |
Perhaps for another ticket, but shouldn't make distclean clean up build artifacts like those subdirectories of build/pkgs/PKG/src/? To test the most recent change, I had to delete the extraneous stuff by hand, or alternatively start with a fresh tarball. Since I'm testing with various branches combined (to allow docbuilding to succeed on my Mac, for example), I would rather not use a fresh tarball in these situations.
Yes, a target sage_docbuild-clean is definitely missing, I'll add one.
Replying to @jhpalmieri:
I would rather not use a fresh tarball in these situations.
What works really well for me is git worktree - easy to create a new "distclean" copy of the branch with this
I get this at the start of doctesting:
ModuleNotFoundError in doctesting framework
**********************************************************************
Traceback (most recent call last):
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.beta7/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 2508, in __call__
doctests, extras = self._run(runner, options, results)
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.beta7/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 2551, in _run
doctests, extras = self.source.create_doctests(sage_namespace)
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.beta7/local/lib/python3.9/site-packages/sage/doctest/sources.py", line 733, in create_doctests
load(filename, namespace) # errors raised here will be caught in DocTestTask
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.beta7/local/lib/python3.9/site-packages/sage/repl/load.py", line 252, in load
exec(code, globals)
File "./clean.py", line 18, in <module>
ModuleNotFoundError: No module named 'sage_setup.find'
leading to
sage -t --long --random-seed=0 src/sage_setup/clean.py # ModuleNotFoundError in doctesting framework
and then I get other doctest failures:
sage -t --long --random-seed=0 src/sage_setup/find.py # 6 doctests failed
sage -t --long --random-seed=0 src/sage_setup/util.py # 2 doctests failed
sage -t --long --random-seed=0 src/sage_setup/optional_extension.py # 2 doctests failed
These all have a similar form:
**********************************************************************
File "src/sage_setup/find.py", line 38, in sage_setup.find.read_distribution
Failed example:
from sage_setup.find import read_distribution
Exception raised:
Traceback (most recent call last):
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.beta7/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 714, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.beta7/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute
exec(compiled, globs)
File "<doctest sage_setup.find.read_distribution[1]>", line 1, in <module>
from sage_setup.find import read_distribution
ModuleNotFoundError: No module named 'sage_setup.find'
**********************************************************************
File "src/sage_setup/find.py", line 92, in sage_setup.find.find_python_sources
Failed example:
from sage_setup.find import find_python_sources
Exception raised:
Traceback (most recent call last):
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.beta7/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 714, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.beta7/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute
exec(compiled, globs)
File "<doctest sage_setup.find.find_python_sources[1]>", line 1, in <module>
from sage_setup.find import find_python_sources
ModuleNotFoundError: No module named 'sage_setup.find'
Branch pushed to git repo; I updated commit sha1. New commits:
1dfeb84 | build/make/Makefile.in (sage_docbuild-clean): New, run it from build-clean |
I don't know how to reproduce the ModuleNotFoundError... could you run make sagelib-clean sagelib and test again?
Replying to @mkoeppe:
I don't know how to reproduce the
ModuleNotFoundError... could you runmake sagelib-clean sageliband test again?
I did that and got the same failures. This was with Sage's own Python and #18272. Then I switched to just this branch and used homebrew's Python and got the same errors.
Branch pushed to git repo; I updated commit sha1. New commits:
881c4af | src/sage_setup/__init__.py: Restore |
Thanks for testing - I've found the problem now. As a leftover from the previous version of the branch, sage_setup had lost its __init__.py file, causing an incomplete installation of it in site-packages
Now the ext directory is missing from build/pkgs/sage_docbuild/src/build/lib/sage_docbuild and similarly for local/lib/python3.9/site-packages/sage_docbuild, so the documentation fails to build. It thinks it succeeds — no error — but the log file says things like
Building ja/a_tour_of_sage.
[a_tour_of] Extension error:
[a_tour_of] Could not import extension sage_docbuild.ext.inventory_builder (exception: No module named 'sage_docbuild.ext')
Build finished. The built documents can be found in /Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.beta7/local/share/doc/sage/html/ja/a_tour_of_sage
Sorry, thanks for your patience with this. I was using find_namespace_packages wrong, now I have just replaced it with an explicit list of packages.
Can you explain the dependencies file?
$(PYTHON) sphinx ../pkgs/sage_docbuild/src/sage_docbuild/*.py ../pkgs/sage_docbuild/src/sage_docbuild/ext/*.py | $(PYTHON_TOOLCHAIN) sagelib
I don't know what .../*.py is supposed to accomplish: how does make know what py files are supposed to be there or what to do with those targets?
These dependencies point to $SAGE_SRC/docbuild (going through the symbolic link at $SAGE_ROOT/build/pkgs/sage_docbuild/src). make expands the wildcard pattern to the source file names there. When they are newer than the timestamp file $SAGE_LOCAL/var/lib/sage/installed/sage_docbuild-9.3.beta7, this triggers a rebuild/reinstallation of the package sage_docbuild.
Okay, thanks, that makes sense. This is working for me now, both building from scratch and upgrading from #31344.
Thank you!
Changed branch from u/mkoeppe/split_sage_setup_docbuild_out_to_a_separate_package to aaab1d3