Move SAGE_ROOT/build/pkgs/*/src to SAGE_ROOT/pkgs/*
mkoeppe opened this issue · 116 comments
In the current layout, these embedded source trees are a little difficult to find / complicated to type.
The proposed location is:
SAGE_ROOT
- pkgs
- sage-conf
- setup.py
- bin/
- sage_conf.py.in
- sage-docbuild
- setup.py
- sage_docbuild -> symlinks to SAGE_ROOT/sage_docbuild/
- sage-sws2rst
- setup.py
- bin/
- sage_sws2rst/
- sagemath-standard
- setup.py
- bin -> symlinks to SAGE_ROOT/src/bin/
- sage -> symlinks to SAGE_ROOT/src/sage/
- sage_setup -> symlinks to SAGE_ROOT/src/sage_setup/ (removed in #29847)
that is, the new directory pkgs will be on the same level as
SAGE_ROOT
- src
the unchanged monolithic sagelib source tree, which will continue to contain subdirectories
- bin/
- doc/
- sage/
- sage_docbuild/ # was sage_setup/docbuild/ in Sage 9.2
- sage_setup/
We add symlinks from the previous locations SAGE_ROOT/build/pkgs/sage[math]_*/src to the new locations SAGE_ROOT/pkgs/sage[math]-*
#29847 will add:
SAGE_ROOT
- pkgs
- sage-setup
- setup.py
- sage_setup -> symlinks to src/sage_setup/
and remove sage_setup from the sagemath-standard distribution.
Other modularization tickets (#29705) will add
SAGE_ROOT
- pkgs
- sagemath-core/
- sagemath-brial/
- sagemath-giac/
- sagemath-meataxe/
- sagemath-tdlib/
etc.
#29868 will add
SAGE_ROOT
- pkgs
- sagemath-doc-html/
- sagemath-doc-pdf/
All files that contain Sage doctests will remain in the monolithic src/ source tree; the source trees of the distributions symlink there. This may be changed in a follow-up ticket regarding the modularization of doctesting.
Before merging this branch into a non-distclean worktree, it is useful to use
git clean -f -x build/pkgs/*/src
CC: @jhpalmieri @dimpase @kiwifb
Component: build
Author: Matthias Koeppe
Branch/Commit: 7568dc6
Reviewer: Dima Pasechnik
Issue created by migration from https://trac.sagemath.org/ticket/31577
Alternatively, these directories could just be put into SAGE_ROOT:
SAGE_ROOT
- sage_conf
- sage_docbuild
- sage_sws2rst
- sagemath_standard
(made this change in the ticket description)
Description changed:
---
+++
@@ -1,14 +1,18 @@
-In the current layout, these embedded source trees are a little difficult to find.
+In the current layout, these embedded source trees are a little difficult to find / complicated to type.
The proposed location is:
```
SAGE_ROOT
- - src
- - pkgs
- - sage_conf
- - sage_docbuild
- - sage_sws2rst
- - sagemath_standard
+ - sage_conf
+ - sage_docbuild
+ - sage_sws2rst
+ - sagemath_standard
+```
+that is, on the same level as
+
+```
+ - src # the monolithic sagelib source tree
```
+Description changed:
---
+++
@@ -12,7 +12,14 @@
that is, on the same level as
```
- - src # the monolithic sagelib source tree
+ - src # the monolithic sagelib source tree - unchanged
+```
+
+Also, we rename the confusingly named directory `SAGE_ROOT/build` (Python users expect a directory named `build` to contains build artifacts only) to
+
+```
+ - sage_bootstrap
```
+Description changed:
---
+++
@@ -5,21 +5,51 @@
```
SAGE_ROOT
- sage_conf
+ - setup.py
+ - sage_conf.py.in
- sage_docbuild
+ - setup.py
+ - sage_docbuild -> symlinks to SAGE_ROOT/src/sage_docbuild
- sage_sws2rst
+ - setup.py
+ - ...
- sagemath_standard
+ - setup.py
+ - sage -> symlinks to SAGE_ROOT/src/sage
+ - sage_setup -> symlinks to SAGE_ROOT/src/sage_setup
```
-that is, on the same level as
+that is, these new directories on the same level as
```
- - src # the monolithic sagelib source tree - unchanged
+ - src
+```
+the unchanged monolithic sagelib source tree, which will continue to contain subdirectories
+
+```
+ - bin
+ - doc
+ - sage
+ - sage_docbuild
+ - sage_setup
```
Also, we rename the confusingly named directory `SAGE_ROOT/build` (Python users expect a directory named `build` to contains build artifacts only) to
```
- sage_bootstrap
+ - setup.py
+ - sage_bootstrap
+ - bin
+ - make
+ - pkgs
+```
+
+#29847 will add:
+
+```
+ - sage_setup
+ - setup.py
+ - sage_setup -> symlinks to SAGE_ROOT/src/sage_setup
```
-Description changed:
---
+++
@@ -18,7 +18,7 @@
- sage -> symlinks to SAGE_ROOT/src/sage
- sage_setup -> symlinks to SAGE_ROOT/src/sage_setup
```
-that is, these new directories on the same level as
+that is, these new directories will be on the same level as
```
- src
@@ -36,20 +36,29 @@
Also, we rename the confusingly named directory `SAGE_ROOT/build` (Python users expect a directory named `build` to contains build artifacts only) to
```
- - sage_bootstrap
- - setup.py
- - sage_bootstrap
- - bin
- - make
- - pkgs
+ - sage_bootstrap
+ - setup.py
+ - sage_bootstrap
+ - bin
+ - make
+ - pkgs
```
#29847 will add:
```
- - sage_setup
- - setup.py
- - sage_setup -> symlinks to SAGE_ROOT/src/sage_setup
+ - sage_setup
+ - setup.py
+ - sage_setup -> symlinks to SAGE_ROOT/src/sage_setup
```
+Other modularization tickets (#29705) will add
+```
+ - sagemath_core
+ - sagemath_brial
+ - sagemath_giac
+ - sagemath_meataxe
+ - sagemath_tdlib
+```
+etc.Description changed:
---
+++
@@ -33,7 +33,7 @@
- sage_setup
```
-Also, we rename the confusingly named directory `SAGE_ROOT/build` (Python users expect a directory named `build` to contains build artifacts only) to
+In follow-up ticket #31662, we rename the confusingly named directory `SAGE_ROOT/build` (Python users expect a directory named `build` to contains build artifacts only) to
```
- sage_bootstrapLast 10 new commits:
0283da5 | build/make/Makefile.in: Add wheel, setuptools_wheel to PYTHON_TOOLCHAIN to make sure that PEP 517 packages have a complete build system |
f720722 | build/pkgs/sagelib/src/tox.ini: Add factor nobuildisolation |
c451b31 | src/setup.cfg.m4 (install_requires): Add sage_conf |
6700223 | Merge tag '9.3.rc0' into t/30913/sagelib__add_setup_cfg__install_requires_ |
04da2c6 | build/pkgs/ipywidgets: Patch out declaring install-requires of nbformat and jupyterlab-widgets |
68dc845 | Merge branch 't/30913/sagelib__add_setup_cfg__install_requires_' into t/31577/move_sage_root_build_pkgs___src_to_sage_root____rename_sage_root_build_to_sage_root_sage_bootstrap |
9181571 | Move /build/pkgs/sage_conf/src to /sage_conf, leave symlink behind |
4f480b4 | Move /build/pkgs/sage_docbuild/src to /sage_docbuild, leave symlink behind |
cc04151 | Move /build/pkgs/sage_sws2rst/src to /sage_sws2rst, leave symlink behind |
8862b55 | Move /build/pkgs/sagelib/src to /sagemath_standard, leave symlink behind |
Author: Matthias Koeppe
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
73d83c2 | Move /build/pkgs/sagelib/src to /sagemath_standard, leave symlink behind |
Branch pushed to git repo; I updated commit sha1. New commits:
ef2fee8 | WIP: Fix up docker build |
sage_docbuild -> symlinks to SAGE_ROOT/src/sage_docbuild
Would it make sense to reverse this: put the files in SAGE_ROOT/sage_docbuild and then make SAGE_ROOT/src/sage_docbuild into a link? Is there any reason to keep SAGE_ROOT/src/sage_docbuild? If so, why also create the top-level directory?
What about SAGE_ROOT/src/sage_setup, along the same lines?
This is a great question.
There is no technical reason for keeping SAGE_ROOT/src/sage_docbuild at all.
Likewise, after #29847, there will be no technical reason for keeping SAGE_ROOT/src/sage_setup.
But in #29705, I promised not to change the structure of the source tree, which I interpret as keeping the monolithic tree SAGE_ROOT/src unchanged. It is so that we don't overwhelm the developer community by making too many changes at the same time.
Of course, it could be argued that sage_docbuild is a new name anyway, and very few people will know or care about this package.
So if you think that's better, I can just move the sage_docbuild sources to SAGE_ROOT/sage_docbuild/sage_docbuild on this ticket.
(I'd rather not put reverse symlinks.)
Branch pushed to git repo; I updated commit sha1. New commits:
12df870 | No more src/sage_docbuild |
Description changed:
---
+++
@@ -9,10 +9,10 @@
- sage_conf.py.in
- sage_docbuild
- setup.py
- - sage_docbuild -> symlinks to SAGE_ROOT/src/sage_docbuild
+ - sage_docbuild # moved here from src/sage_docbuild - in Sage 9.2 was src/sage_setup/docbuild
- sage_sws2rst
- setup.py
- - ...
+ - sage_sws2rst
- sagemath_standard
- setup.py
- sage -> symlinks to SAGE_ROOT/src/sage
@@ -49,7 +49,7 @@
```
- sage_setup
- setup.py
- - sage_setup -> symlinks to SAGE_ROOT/src/sage_setup
+ - sage_setup # SAGE_ROOT/src/sage_setup will be moved here
```
Other modularization tickets (#29705) will addDescription changed:
---
+++
@@ -23,13 +23,12 @@
```
- src
```
-the unchanged monolithic sagelib source tree, which will continue to contain subdirectories
+the monolithic sagelib source tree, unchanged compared to Sage 9.2, which will continue to contain subdirectories
```
- bin
- doc
- sage
- - sage_docbuild
- sage_setup
```
I will make another change, renaming the top-level directories from using underscores to using dashes. This will emphasize that these are the names of distribution packages (PyPI normalizes to dashes).
Description changed:
---
+++
@@ -4,16 +4,16 @@
```
SAGE_ROOT
- - sage_conf
+ - sage-conf
- setup.py
- sage_conf.py.in
- - sage_docbuild
+ - sage-docbuild
- setup.py
- sage_docbuild # moved here from src/sage_docbuild - in Sage 9.2 was src/sage_setup/docbuild
- - sage_sws2rst
+ - sage-sws2rst
- setup.py
- sage_sws2rst
- - sagemath_standard
+ - sagemath-standard
- setup.py
- sage -> symlinks to SAGE_ROOT/src/sage
- sage_setup -> symlinks to SAGE_ROOT/src/sage_setup
@@ -35,7 +35,7 @@
In follow-up ticket #31662, we rename the confusingly named directory `SAGE_ROOT/build` (Python users expect a directory named `build` to contains build artifacts only) to
```
- - sage_bootstrap
+ - sage-bootstrap
- setup.py
- sage_bootstrap
- bin
@@ -46,7 +46,7 @@
#29847 will add:
```
- - sage_setup
+ - sage-setup
- setup.py
- sage_setup # SAGE_ROOT/src/sage_setup will be moved here
```
@@ -54,10 +54,11 @@
Other modularization tickets (#29705) will add
```
- - sagemath_core
- - sagemath_brial
- - sagemath_giac
- - sagemath_meataxe
- - sagemath_tdlib
+ - sagemath-core
+ - sagemath-brial
+ - sagemath-giac
+ - sagemath-meataxe
+ - sagemath-tdlib
```
etc.
+(This change is making the kludge that I had to add to the top-level Makefile unnecessary. The command for building make sage_conf, will no longer be fooled by the existence of a top-level directory of that name.)
Branch pushed to git repo; I updated commit sha1. New commits:
1fe5f3d | Spell distribution source directories sage-conf, sage-docbuild, sage-sws2rst, sagemath-standard with dashes, not underscores |
Description changed:
---
+++
@@ -6,17 +6,20 @@
SAGE_ROOT
- sage-conf
- setup.py
+ - bin/
- sage_conf.py.in
- sage-docbuild
- setup.py
- - sage_docbuild # moved here from src/sage_docbuild - in Sage 9.2 was src/sage_setup/docbuild
+ - sage_docbuild # moved here from src/sage_docbuild/ - in Sage 9.2 was src/sage_setup/docbuild/
- sage-sws2rst
- setup.py
- - sage_sws2rst
+ - bin/
+ - sage_sws2rst/
- sagemath-standard
- setup.py
- - sage -> symlinks to SAGE_ROOT/src/sage
- - sage_setup -> symlinks to SAGE_ROOT/src/sage_setup
+ - bin -> symlinks to SAGE_ROOT/src/bin/
+ - sage -> symlinks to SAGE_ROOT/src/sage/
+ - sage_setup -> symlinks to SAGE_ROOT/src/sage_setup/
```
that is, these new directories will be on the same level as
@@ -26,10 +29,10 @@
the monolithic sagelib source tree, unchanged compared to Sage 9.2, which will continue to contain subdirectories
```
- - bin
- - doc
- - sage
- - sage_setup
+ - bin/
+ - doc/
+ - sage/
+ - sage_setup/
```
In follow-up ticket #31662, we rename the confusingly named directory `SAGE_ROOT/build` (Python users expect a directory named `build` to contains build artifacts only) to
@@ -37,10 +40,10 @@
```
- sage-bootstrap
- setup.py
- - sage_bootstrap
- - bin
- - make
- - pkgs
+ - sage_bootstrap/
+ - bin/
+ - make/
+ - pkgs/
```
#29847 will add:
@@ -48,17 +51,17 @@
```
- sage-setup
- setup.py
- - sage_setup # SAGE_ROOT/src/sage_setup will be moved here
+ - sage_setup/ # SAGE_ROOT/src/sage_setup will be moved here
```
Other modularization tickets (#29705) will add
```
- - sagemath-core
- - sagemath-brial
- - sagemath-giac
- - sagemath-meataxe
- - sagemath-tdlib
+ - sagemath-core/
+ - sagemath-brial/
+ - sagemath-giac/
+ - sagemath-meataxe/
+ - sagemath-tdlib/
```
etc.
Description changed:
---
+++
@@ -10,7 +10,7 @@
- sage_conf.py.in
- sage-docbuild
- setup.py
- - sage_docbuild # moved here from src/sage_docbuild/ - in Sage 9.2 was src/sage_setup/docbuild/
+ - sage_docbuild/ # moved here from src/sage_docbuild/ - in Sage 9.2 was src/sage_setup/docbuild/
- sage-sws2rst
- setup.py
- bin/Branch pushed to git repo; I updated commit sha1. New commits:
8ad9b3e | Fixup renames |
Branch pushed to git repo; I updated commit sha1. New commits:
4c0d38d | Also standardize on using dashes in distribution names in setup.cfg |
Description changed:
---
+++
@@ -65,3 +65,10 @@
```
etc.
+#29868 will add
+
+```
+ - sagemath-doc-html/
+ - sagemath-doc-pdf/
+```
+Branch pushed to git repo; I updated commit sha1. New commits:
47b890e | Merge tag '9.3.rc3' into t/31577/move_sage_root_build_pkgs___src_to_sage_root____rename_sage_root_build_to_sage_root_sage_bootstrap |
make ptestlong fails to run any doctests because of this error:
Using --optional=build,dochtml,homebrew,pip,sage,sage_spkg
Doctesting entire Sage library.
Traceback (most recent call last):
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc2/src/bin/sage-runtests", line 144, in <module>
err = DC.run()
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc2/local/lib/python3.9/site-packages/sage/doctest/control.py", line 1204, in run
self.expand_files_into_sources()
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc2/local/lib/python3.9/site-packages/sage/doctest/control.py", line 788, in expand_files_into_sources
self.sources = [FileDocTestSource(path, self.options) for path in expand()]
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc2/local/lib/python3.9/site-packages/sage/doctest/control.py", line 788, in <listcomp>
self.sources = [FileDocTestSource(path, self.options) for path in expand()]
File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc2/local/lib/python3.9/site-packages/sage/doctest/sources.py", line 528, in __init__
raise ValueError("unknown file extension %r"%ext)
ValueError: unknown file extension ''
make: *** [ptestlong] Error 1
I believe the problem is that it trying to doctest src/sage_docbuild.
OK, I guess I have to retract my claim that there is no technical reason for having src/sage_docbuild. In part that's #31352 though
I'm currently testing this change:
diff --git a/src/sage/doctest/control.py b/src/sage/doctest/control.py
index afb0d6cbcf..45ec2263fb 100644
--- a/src/sage/doctest/control.py
+++ b/src/sage/doctest/control.py
@@ -695,7 +695,7 @@ class DocTestController(SageObject):
# don't make sense to run outside a build environment
if have_git:
self.files.append(opj(SAGE_SRC, 'sage_setup'))
- self.files.append(opj(SAGE_SRC, 'sage_docbuild'))
+ self.files.append(opj(SAGE_ROOT, 'sage-docbuild'))
self.files.append(SAGE_DOC_SRC)
if self.options.all or (self.options.new and not have_git):I think I'd rather undo the change in 12df870 and keep it for another ticket.
I think it's better to keep everything that is subject to Sage doctesting in one physical tree
(Modularization of doctesting is a separate issue.)
Replying to @jhpalmieri:
I'm currently testing this change:
diff --git a/src/sage/doctest/control.py b/src/sage/doctest/control.py index afb0d6cbcf..45ec2263fb 100644 --- a/src/sage/doctest/control.py +++ b/src/sage/doctest/control.py @@ -695,7 +695,7 @@ class DocTestController(SageObject): # don't make sense to run outside a build environment if have_git: self.files.append(opj(SAGE_SRC, 'sage_setup')) - self.files.append(opj(SAGE_SRC, 'sage_docbuild')) + self.files.append(opj(SAGE_ROOT, 'sage-docbuild')) self.files.append(SAGE_DOC_SRC) if self.options.all or (self.options.new and not have_git):
I really don't like that kind of changes. In sage-on-distro SAGE_SRC becomes SAGE_LIB at runtime when SAGE_ROOT is undefined. That would break things there. Not to mention I am against introducing more SAGE_ROOT outside of the build system as a matter of principle. As Matthias says modularization of doctesting is a separate issue and that's what would be most appropriate there.
Replying to @kiwifb:
Replying to @jhpalmieri:
I'm currently testing this change:
diff --git a/src/sage/doctest/control.py b/src/sage/doctest/control.py index afb0d6cbcf..45ec2263fb 100644 --- a/src/sage/doctest/control.py +++ b/src/sage/doctest/control.py @@ -695,7 +695,7 @@ class DocTestController(SageObject): # don't make sense to run outside a build environment if have_git: self.files.append(opj(SAGE_SRC, 'sage_setup')) - self.files.append(opj(SAGE_SRC, 'sage_docbuild')) + self.files.append(opj(SAGE_ROOT, 'sage-docbuild')) self.files.append(SAGE_DOC_SRC) if self.options.all or (self.options.new and not have_git):I really don't like that kind of changes. In sage-on-distro
SAGE_SRCbecomesSAGE_LIBat runtime whenSAGE_ROOTis undefined. That would break things there. Not to mention I am against introducing moreSAGE_ROOToutside of the build system as a matter of principle. As Matthias says modularization of doctesting is a separate issue and that's what would be most appropriate there.
I think that on distros, have_git will be false (it tests whether SAGE_ROOT/.git exists), so this block, and the resulting doctests, will be skipped entirely.
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
a24fe5c | Merge #30913 |
ca9935e | Move /build/pkgs/sage_conf/src to /sage_conf, leave symlink behind |
f81216a | Move /build/pkgs/sage_docbuild/src to /sage_docbuild, leave symlink behind |
b5e30f4 | Move /build/pkgs/sage_sws2rst/src to /sage_sws2rst, leave symlink behind |
25e6d14 | Move /build/pkgs/sagelib/src to /sagemath_standard, leave symlink behind |
e253a2b | .gitignore: Update for new source tree locations |
1cfeb06 | Spell distribution source directories sage-conf, sage-docbuild, sage-sws2rst, sagemath-standard with dashes, not underscores |
8d4ea4e | Also standardize on using dashes in distribution names in setup.cfg |
08b8306 | WIP: Fix up docker build |
You are right, it will skipped. Although technically they could be run if present.
Nevertheless I have removed that change. Too complicated for 1 ticket.
Description changed:
---
+++
@@ -10,7 +10,7 @@
- sage_conf.py.in
- sage-docbuild
- setup.py
- - sage_docbuild/ # moved here from src/sage_docbuild/ - in Sage 9.2 was src/sage_setup/docbuild/
+ - sage_docbuild -> symlinks to SAGE_ROOT/sage_docbuild/
- sage-sws2rst
- setup.py
- bin/
@@ -26,12 +26,13 @@
```
- src
```
-the monolithic sagelib source tree, unchanged compared to Sage 9.2, which will continue to contain subdirectories
+the unchanged monolithic sagelib source tree, which will continue to contain subdirectories
```
- bin/
- doc/
- sage/
+ - sage_docbuild/ # was sage_setup/docbuild/ in Sage 9.2
- sage_setup/
```
@@ -51,7 +52,7 @@
```
- sage-setup
- setup.py
- - sage_setup/ # SAGE_ROOT/src/sage_setup will be moved here
+ - sage_setup -> symlinks to src/sage_setup/
```
Other modularization tickets (#29705) will add
@@ -72,3 +73,7 @@
- sagemath-doc-pdf/
```
+All files that contain Sage doctests will remain in the monolithic `src/` source tree; the source trees of the distributions symlink there. This may be changed in a follow-up ticket regarding the modularization of doctesting.
+
+
+Replying to @kiwifb:
You are right, it will skipped. Although technically they could be run if present.
One of the goals of that particular method is to exclude files if in a distro. I don't know enough about distros to know if the method is well-implemented, though. (I didn't write it.)
Some related questions about this ticket: how does this new organization affect distros, and should we be doctesting any of the pieces that are being moved to SAGE_ROOT?
Replying to @jhpalmieri:
should we be doctesting any of the pieces that are being moved to
SAGE_ROOT?
sage-sws2rst has its own simple test script, not related to the Sage doctester.
The other packages do not have separate tests but are doctested as part of the Sage.
The assumption is here that the doctester is run after all of these packages are installed. This assumption would continue to hold even as the modularization goes forward. I don't really have a plan for modularizing the doctesting, other than #30778.
Replying to @jhpalmieri:
how does this new organization affect distros
I would hope, not at all. All paths that were valid before are still valid afterwards thanks to the symlinks build/pkgs/SOME_SPKG/src -> /SOME-SPKG
Description changed:
---
+++
@@ -35,6 +35,8 @@
- sage_docbuild/ # was sage_setup/docbuild/ in Sage 9.2
- sage_setup/
```
+
+We add symlinks from the previous locations `SAGE_ROOT/build/pkgs/sage[math]_*/src` to the new locations `SAGE_ROOT/sage[math]-*`
In follow-up ticket #31662, we rename the confusingly named directory `SAGE_ROOT/build` (Python users expect a directory named `build` to contains build artifacts only) to
Description changed:
---
+++
@@ -56,6 +56,7 @@
- setup.py
- sage_setup -> symlinks to src/sage_setup/
```
+and remove `sage_setup` from the `sagemath-standard` distribution.
Other modularization tickets (#29705) will add
Description changed:
---
+++
@@ -19,7 +19,7 @@
- setup.py
- bin -> symlinks to SAGE_ROOT/src/bin/
- sage -> symlinks to SAGE_ROOT/src/sage/
- - sage_setup -> symlinks to SAGE_ROOT/src/sage_setup/
+ - sage_setup -> symlinks to SAGE_ROOT/src/sage_setup/ (removed in #29847)
```
that is, these new directories will be on the same level as
Replying to @mkoeppe:
Replying to @jhpalmieri:
how does this new organization affect distros
I would hope, not at all. All paths that were valid before are still valid afterwards thanks to the symlinks
build/pkgs/SOME_SPKG/src -> /SOME-SPKG
As far as I am concerned, anything living under build/pkgs/ should have a separate tarball and be installed as a separate package. This has been somewhat challenged by the way Matthias is conducting the modularization. But that's the goal at the end of the day. If it is there it is not supposed to be anything special anymore.
That being said sage_conf is being a pain because it is still very special as far as I can see. It still need a real SAGE_ROOT tree. But that is a different conversation and ticket.
That's right, after this ticket, SAGE_ROOT/sage-docbuild, SAGE_ROOT/sage-sws2rst, SAGE_ROOT/sagemath-standard, and SAGE_ROOT/sage-conf are standard source trees of Python distribution packages, containing nothing that is specific to Sage-the-distribution.
It is possible to build sdists from each of these source trees using setup.py sdist, and wheels using setup.py bdist_wheel.
Actually also sage-conf has the same status: You can build an sdist or a wheel from it. However, this version of sage-conf is not useful for putting it on PyPI. #29039 and #31396 create separate versions of sage-conf that are suitable for distribution.
Replying to @mkoeppe:
Actually also
sage-confhas the same status: You can build an sdist or a wheel from it. However, this version ofsage-confis not useful for putting it on PyPI. #29039 and #31396 create separate versions ofsage-confthat are suitable for distribution.
That's right, I had to remove a lot of stuff from the PyPI version before I could make it do the bits that would be useful to me properly as someone who build stuff from source. Considering the amount of stuff I just want to throw out, it is almost a different package altogether.
Yes, downstream packaging is encouraged to write a separate sage_conf for their purposes.
Replying to @mkoeppe:
Yes, downstream packaging is encouraged to write a separate
sage_conffor their purposes.
Beurk. Anyway, we should have another meeting may be the next sage-days. I think there are things we should do to the design of "those" configuration files at large, and that could make it easier for distros in the process. I need to figure if what I think of who be easily doable python side. And it is definitely a separate conversation from this ticket.
It's really very easy:
- You want to run our
configure-> use one of our files which comes from configure. - You don't want to run
configure-> write yoursage_conf.pyin which you hardcode your stuff.
Replying to @mkoeppe:
It's really very easy:
- You don't want to run
configure-> write yoursage_conf.pyin which you hardcode your stuff.
This is probably the best solution and it is also incredibly dirty once you get QA and multi-python support involved. In short this is cheap and easy if your packaging is somewhat ad hoc. Once you want this to go inside a framework with rules, there is a higher price than you may expect.
But we are getting incredibly off-topic and should stop polluting this ticket.
Yes, let's continue with this at another time and on another ticket.
% ./sage -t src/sage/misc/package.py
Running doctests with ID 2021-04-12-22-39-35-af3c76ef.
Git branch: t/31577/move_sage_root_build_pkgs___src_to_sage_root____rename_sage_root_build_to_sage_root_sage_bootstrap
Using --optional=build,dochtml,homebrew,pip,sage,sage_spkg
Doctesting 1 file.
sage -t --warn-long 125.4 --random-seed=0 src/sage/misc/package.py
**********************************************************************
File "src/sage/misc/package.py", line 336, in sage.misc.package.installed_packages
Failed example:
installed_packages()['sage_conf'] # optional - build
Expected:
'none'
Got:
'9.3.rc3'
**********************************************************************
1 item had failures:
1 of 4 in sage.misc.package.installed_packages
[50 tests, 1 failure, 2.86 s]
----------------------------------------------------------------------
sage -t --warn-long 125.4 --random-seed=0 src/sage/misc/package.py # 1 doctest failed
----------------------------------------------------------------------
Branch pushed to git repo; I updated commit sha1. New commits:
248fae7 | src/sage/misc/package.py: sage_conf now has a version |
Thanks for testing!
Work Issues: Fix up "tox -e docker..."
What are the files Pipfile, Pipfile.m4, etc. for? I just build Sage, and for example, sagemath-standard/Pipfile is currently a link pointing to a nonexistent target, and the same for sagemath-standard/Pipfile-dist.
If you run ./bootstrap, the targets of these symlinks should be generated.
These files were introduced in #30913. They are not used in a normal build. They are for users who want to use https://pypi.org/project/pipenv/ to set up a virtual environment.
I might prefer this in place of, or in addition to, the fixed installed_packages() doctest:
diff --git a/src/sage/misc/package.py b/src/sage/misc/package.py
index e402293baf..f9a3f9fc9e 100644
--- a/src/sage/misc/package.py
+++ b/src/sage/misc/package.py
@@ -335,6 +335,8 @@ def installed_packages(exclude_pip=True):
'0.7.12'
sage: installed_packages()['sage_conf'] # optional - build, random
'9.3'
+ sage: installed_packages()['sage_conf'] == sage.env.SAGE_VERSION # optional - build
+ True
.. SEEALSO::
That's a good idea
Branch pushed to git repo; I updated commit sha1. New commits:
58bee8e | src/sage/misc/package.py: Better doctest for installed_packages |
Branch pushed to git repo; I updated commit sha1. New commits:
16b54d1 | build/pkgs/{sage_docbuild,sage_sws2rst}/package-version.txt: Symlink to ../sagelib/package-version.txt; do not symlink into src to make things easier for the Docker build |
Unfortunately I have a bit of trouble to make this layout work well with the tox-docker build.
How about pushing the new directories down one level, to something like SAGE_ROOT/pkgs/sage-conf etc.?
Should the Sage source be in the same place? Or rather, should everything be in src? I don't know if there is any standard way to do this or any particular logic behind it.
I don't think there is a standard for this.
I initially had made it SAGE_ROOT/src/pkgs/sage-conf but I think it would be cleaner if src only contained the monolithic Sage library.
Then sure, SAGE_ROOT/pkgs/sage-conf sounds good.
Description changed:
---
+++
@@ -4,6 +4,7 @@
```
SAGE_ROOT
+- pkgs
- sage-conf
- setup.py
- bin/
@@ -21,33 +22,22 @@
- sage -> symlinks to SAGE_ROOT/src/sage/
- sage_setup -> symlinks to SAGE_ROOT/src/sage_setup/ (removed in #29847)
```
-that is, these new directories will be on the same level as
+that is, the new directory `pkgs` will be on the same level as
```
- - src
+- src
```
the unchanged monolithic sagelib source tree, which will continue to contain subdirectories
```
- - bin/
- - doc/
- - sage/
- - sage_docbuild/ # was sage_setup/docbuild/ in Sage 9.2
- - sage_setup/
+ - bin/
+ - doc/
+ - sage/
+ - sage_docbuild/ # was sage_setup/docbuild/ in Sage 9.2
+ - sage_setup/
```
We add symlinks from the previous locations `SAGE_ROOT/build/pkgs/sage[math]_*/src` to the new locations `SAGE_ROOT/sage[math]-*`
-
-In follow-up ticket #31662, we rename the confusingly named directory `SAGE_ROOT/build` (Python users expect a directory named `build` to contains build artifacts only) to
-
-```
- - sage-bootstrap
- - setup.py
- - sage_bootstrap/
- - bin/
- - make/
- - pkgs/
-```
#29847 will add:
Changed work issues from Fix up "tox -e docker..." to Make it SAGE_ROOT/pkgs/...
Description changed:
---
+++
@@ -42,6 +42,8 @@
#29847 will add:
```
+SAGE_ROOT
+- pkgs
- sage-setup
- setup.py
- sage_setup -> symlinks to src/sage_setup/
@@ -51,6 +53,8 @@
Other modularization tickets (#29705) will add
```
+SAGE_ROOT
+- pkgs
- sagemath-core/
- sagemath-brial/
- sagemath-giac/
@@ -62,6 +66,8 @@
#29868 will add
```
+SAGE_ROOT
+- pkgs
- sagemath-doc-html/
- sagemath-doc-pdf/
```Description changed:
---
+++
@@ -25,6 +25,7 @@
that is, the new directory `pkgs` will be on the same level as
```
+SAGE_ROOT
- src
```
the unchanged monolithic sagelib source tree, which will continue to contain subdirectoriesChanged work issues from Make it SAGE_ROOT/pkgs/... to Make it SAGE_ROOT/pkgs/..., merge #31825
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
9c3b887 | Move /build/pkgs/sage_docbuild/src to /sage_docbuild, leave symlink behind |
1628441 | Move /build/pkgs/sage_sws2rst/src to /sage_sws2rst, leave symlink behind |
a5d164b | Move /build/pkgs/sagelib/src to /sagemath_standard, leave symlink behind |
673de88 | .gitignore: Update for new source tree locations |
2bdd5bd | Spell distribution source directories sage-conf, sage-docbuild, sage-sws2rst, sagemath-standard with dashes, not underscores |
e35a6d1 | Also standardize on using dashes in distribution names in setup.cfg |
89c78e8 | WIP: Fix up docker build |
fac294d | build/pkgs/{sage_docbuild,sage_sws2rst}/package-version.txt: Symlink to ../sagelib/package-version.txt; do not symlink into src to make things easier for the Docker build |
f0f5d54 | build/bin/write-dockerfile.sh: Adjust ADD commands |
6ad0b8f | WIP update .gitignore/.dockerignore |
Branch pushed to git repo; I updated commit sha1. New commits:
c58e51b | Move package source directories to pkgs/ |
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
ce3c4d3 | Move /build/pkgs/sage_conf/src to /sage_conf, leave symlink behind |
7824e0b | Also standardize on using dashes in distribution names in setup.cfg |
49323d2 | build/pkgs/{sage_docbuild,sage_sws2rst}/package-version.txt: Symlink to ../sagelib/package-version.txt; do not symlink into src to make things easier for the Docker build |
1d23923 | update .gitignore/.dockerignore; Move package source directories to pkgs/ |
Description changed:
---
+++
@@ -38,7 +38,7 @@
- sage_setup/
```
-We add symlinks from the previous locations `SAGE_ROOT/build/pkgs/sage[math]_*/src` to the new locations `SAGE_ROOT/sage[math]-*`
+We add symlinks from the previous locations `SAGE_ROOT/build/pkgs/sage[math]_*/src` to the new locations `SAGE_ROOT/pkgs/sage[math]-*`
#29847 will add:
@@ -73,7 +73,7 @@
- sagemath-doc-pdf/
```
-All files that contain Sage doctests will remain in the monolithic `src/` source tree; the source trees of the distributions symlink there. This may be changed in a follow-up ticket regarding the modularization of doctesting.
+**All files that contain Sage doctests will remain in the monolithic `src/` source tree;** the source trees of the distributions symlink there. This may be changed in a follow-up ticket regarding the modularization of doctesting.
Branch pushed to git repo; I updated commit sha1. New commits:
0cda8f1 | README: Update directory chart |